From patchwork Mon May 24 12:53:03 2010 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gautham R Shenoy X-Patchwork-Id: 53413 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [199.232.76.165]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by ozlabs.org (Postfix) with ESMTPS id 0B5A7B7D20 for ; Mon, 24 May 2010 22:56:54 +1000 (EST) Received: from localhost ([127.0.0.1]:57959 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OGXD7-0005xq-Cq for incoming@patchwork.ozlabs.org; Mon, 24 May 2010 08:56:49 -0400 Received: from [140.186.70.92] (port=44411 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OGX9e-0004Xm-7B for qemu-devel@nongnu.org; Mon, 24 May 2010 08:53:17 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OGX9b-00070o-M9 for qemu-devel@nongnu.org; Mon, 24 May 2010 08:53:13 -0400 Received: from e23smtp03.au.ibm.com ([202.81.31.145]:50370) by eggs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1OGX9a-000704-TY for qemu-devel@nongnu.org; Mon, 24 May 2010 08:53:11 -0400 Received: from d23relay05.au.ibm.com (d23relay05.au.ibm.com [202.81.31.247]) by e23smtp03.au.ibm.com (8.14.3/8.13.1) with ESMTP id o4OCnePN011572 for ; Mon, 24 May 2010 22:49:40 +1000 Received: from d23av04.au.ibm.com (d23av04.au.ibm.com [9.190.235.139]) by d23relay05.au.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id o4OCr4Ne1294524 for ; Mon, 24 May 2010 22:53:04 +1000 Received: from d23av04.au.ibm.com (loopback [127.0.0.1]) by d23av04.au.ibm.com (8.14.3/8.13.1/NCO v10.0 AVout) with ESMTP id o4OCr4LF028034 for ; Mon, 24 May 2010 22:53:04 +1000 Received: from sofia.in.ibm.com ([9.124.35.40]) by d23av04.au.ibm.com (8.14.3/8.13.1/NCO v10.0 AVin) with ESMTP id o4OCr3bT028020; Mon, 24 May 2010 22:53:03 +1000 Received: from localhost.localdomain (localhost [IPv6:::1]) by sofia.in.ibm.com (Postfix) with ESMTP id 2D9B7E4B35; Mon, 24 May 2010 18:23:03 +0530 (IST) To: Qemu-development List From: Gautham R Shenoy Date: Mon, 24 May 2010 18:23:03 +0530 Message-ID: <20100524125303.29646.8967.stgit@localhost.localdomain> In-Reply-To: <20100524125220.29646.62160.stgit@localhost.localdomain> References: <20100524125220.29646.62160.stgit@localhost.localdomain> User-Agent: StGit/0.15-51-gc750 MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6, seldom 2.4 (older, 4) Cc: Anthony Liguori , Avi Kivity , "Aneesh Kumar K.V" Subject: [Qemu-devel] [RFC/ PATCH 1/4] qemu: Generic asynchronous threading framework to offload tasks X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: Aneesh Kumar K.V This patch creates a generic asynchronous-task-offloading infrastructure. It's extracted out of the threading framework that is being used by paio. The reason for extracting out this generic infrastructure of the posix-aio-compat.c is so that other subsystems, such as virtio-9p could make use of it for offloading tasks that could block. [ego@in.ibm.com: work_item_pool, async_work_init, async_work_release, async_cancel_work] Signed-off-by: Aneesh Kumar K.V Signed-off-by: Gautham R Shenoy --- Makefile.objs | 2 + async-work.c | 152 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ async-work.h | 85 ++++++++++++++++++++++++++++++++ 3 files changed, 239 insertions(+), 0 deletions(-) create mode 100644 async-work.c create mode 100644 async-work.h diff --git a/Makefile.objs b/Makefile.objs index ecdd53e..faf3d67 100644 --- a/Makefile.objs +++ b/Makefile.objs @@ -9,6 +9,7 @@ qobject-obj-y += qerror.o block-obj-y = cutils.o cache-utils.o qemu-malloc.o qemu-option.o module.o block-obj-y += nbd.o block.o aio.o aes.o osdep.o qemu-config.o +block-obj-y += async-work.o block-obj-$(CONFIG_POSIX) += posix-aio-compat.o block-obj-$(CONFIG_LINUX_AIO) += linux-aio.o @@ -44,6 +45,7 @@ fsdev-obj-$(CONFIG_LINUX) += $(addprefix fsdev/, $(fsdev-nested-y)) # system emulation, i.e. a single QEMU executable should support all # CPUs and machines. +#common-obj-y = $(asyncwork-obj-y) common-obj-y = $(block-obj-y) common-obj-y += $(net-obj-y) common-obj-y += $(qobject-obj-y) diff --git a/async-work.c b/async-work.c new file mode 100644 index 0000000..5195bbf --- /dev/null +++ b/async-work.c @@ -0,0 +1,152 @@ +/* + * Async work support + * + * Copyright IBM, Corp. 2010 + * + * Authors: + * Aneesh Kumar K.V + * + * This work is licensed under the terms of the GNU GPL, version 2. See + * the COPYING file in the top-level directory. + * + */ +#include +#include +#include +#include +#include +#include "async-work.h" +#include "osdep.h" + +static void async_abort(int err, const char *what) +{ + fprintf(stderr, "%s failed: %s\n", what, strerror(err)); + abort(); +} + +static void *async_worker_thread(void *data) +{ + struct async_queue *queue = data; + + while (1) { + struct work_item *work; + int ret = 0; + qemu_timeval tv; + struct timespec ts; + + qemu_gettimeofday(&tv); + ts.tv_sec = tv.tv_sec + 10; + ts.tv_nsec = 0; + + pthread_mutex_lock(&(queue->lock)); + + while (QTAILQ_EMPTY(&(queue->request_list)) && + (ret != ETIMEDOUT)) { + ret = pthread_cond_timedwait(&(queue->cond), + &(queue->lock), &ts); + } + + if (QTAILQ_EMPTY(&(queue->request_list))) + goto check_exit; + + work = QTAILQ_FIRST(&(queue->request_list)); + QTAILQ_REMOVE(&(queue->request_list), work, node); + queue->idle_threads--; + pthread_mutex_unlock(&(queue->lock)); + + /* execute the work function */ + work->func(work); + async_work_release(queue, work); + + pthread_mutex_lock(&(queue->lock)); + queue->idle_threads++; + +check_exit: + if ((queue->idle_threads > 0) && + (queue->cur_threads > queue->min_threads)) { + /* we retain minimum number of threads */ + break; + } + pthread_mutex_unlock(&(queue->lock)); + } + + queue->idle_threads--; + queue->cur_threads--; + pthread_mutex_unlock(&(queue->lock)); + + return NULL; +} + +static void spawn_async_thread(struct async_queue *queue) +{ + int ret; + pthread_attr_t attr; + pthread_t thread_id; + sigset_t set, oldset; + + queue->cur_threads++; + queue->idle_threads++; + ret = pthread_attr_init(&attr); + if (ret) { + async_abort(ret, "pthread_attr_init"); + } + + /* create a detached thread so that we don't need to wait on it */ + ret = pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED); + if (ret) { + async_abort(ret, "pthread_attr_setdetachstate"); + } + + /* block all signals */ + if (sigfillset(&set)) { + async_abort(errno, "sigfillset"); + } + + if (sigprocmask(SIG_SETMASK, &set, &oldset)) { + async_abort(errno, "sigprocmask"); + } + + ret = pthread_create(&thread_id, &attr, async_worker_thread, queue); + if (ret) { + async_abort(ret, "pthread_create"); + } + + if (sigprocmask(SIG_SETMASK, &oldset, NULL)) { + async_abort(errno, "sigprocmask restore"); + } +} + +void qemu_async_submit(struct async_queue *queue, struct work_item *work) +{ + pthread_mutex_lock(&(queue->lock)); + if (queue->idle_threads == 0 && queue->cur_threads < queue->max_threads) { + spawn_async_thread(queue); + } + QTAILQ_INSERT_TAIL(&(queue->request_list), work, node); + pthread_mutex_unlock(&(queue->lock)); + pthread_cond_signal(&(queue->cond)); +} + +int qemu_async_cancel_work(struct async_queue *queue, struct work_item *work) +{ + struct work_item *ret_work; + int found = 0; + + pthread_mutex_lock(&(queue->lock)); + QTAILQ_FOREACH(ret_work, &(queue->request_list), node) { + if (ret_work == work) { + QTAILQ_REMOVE(&(queue->request_list), ret_work, node); + found = 1; + break; + } + } + pthread_mutex_unlock(&(queue->lock)); + + if (found) { + async_work_release(queue, work); + return 0; + } + + return 1; +} + diff --git a/async-work.h b/async-work.h new file mode 100644 index 0000000..eef60f7 --- /dev/null +++ b/async-work.h @@ -0,0 +1,85 @@ +/* + * Async work support + * + * Copyright IBM, Corp. 2010 + * + * Authors: + * Aneesh Kumar K.V + * + * This work is licensed under the terms of the GNU GPL, version 2. See + * the COPYING file in the top-level directory. + * + */ +#ifndef QEMU_ASYNC_WORK_H +#define QEMU_ASYNC_WORK_H + +#include +#include "qemu-queue.h" +#include "qemu-common.h" + +struct async_queue +{ + pthread_mutex_t lock; + pthread_cond_t cond; + int max_threads; + int min_threads; + int cur_threads; + int idle_threads; + QTAILQ_HEAD(, work_item) request_list; + QTAILQ_HEAD(, work_item) work_item_pool; +}; + +struct work_item +{ + QTAILQ_ENTRY(work_item) node; + void (*func)(struct work_item *work); + void *private; +}; + +static inline void async_queue_init(struct async_queue *queue, + int max_threads, int min_threads) +{ + queue->cur_threads = 0; + queue->idle_threads = 0; + queue->max_threads = max_threads; + queue->min_threads = min_threads; + QTAILQ_INIT(&(queue->request_list)); + QTAILQ_INIT(&(queue->work_item_pool)); + pthread_mutex_init(&(queue->lock), NULL); + pthread_cond_init(&(queue->cond), NULL); +} + +static inline struct work_item *async_work_init(struct async_queue *queue, + void (*func)(struct work_item *), + void *data) +{ + struct work_item *work; + pthread_mutex_lock(&(queue->lock)); + if (QTAILQ_EMPTY(&(queue->work_item_pool))) { + work = qemu_mallocz(sizeof(*work)); + } else { + work = QTAILQ_FIRST(&(queue->work_item_pool)); + QTAILQ_REMOVE(&(queue->work_item_pool), work, node); + } + + work->func = func; + work->private = data; + pthread_mutex_unlock(&(queue->lock)); + + return work; +} + +static inline void async_work_release(struct async_queue *queue, + struct work_item *work) +{ + pthread_mutex_lock(&(queue->lock)); + QTAILQ_INSERT_TAIL(&(queue->work_item_pool), work, node); + pthread_mutex_unlock(&(queue->lock)); +} + +extern void qemu_async_submit(struct async_queue *queue, + struct work_item *work); + +extern int qemu_async_cancel_work(struct async_queue *queue, + struct work_item *work); +#endif