From patchwork Fri Mar 9 16:02:23 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Sementsov-Ogievskiy X-Patchwork-Id: 883730 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=virtuozzo.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zyXGv4LXKz9sc2 for ; Sat, 10 Mar 2018 03:03:19 +1100 (AEDT) Received: from localhost ([::1]:46157 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euKUH-0006bU-GU for incoming@patchwork.ozlabs.org; Fri, 09 Mar 2018 11:03:17 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56646) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euKTZ-0006Yt-Lc for qemu-devel@nongnu.org; Fri, 09 Mar 2018 11:02:40 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1euKTU-0005yp-QP for qemu-devel@nongnu.org; Fri, 09 Mar 2018 11:02:33 -0500 Received: from relay.sw.ru ([185.231.240.75]:40034) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1euKTU-0005xx-IH; Fri, 09 Mar 2018 11:02:28 -0500 Received: from msk-vpn.virtuozzo.com ([195.214.232.6] helo=kvm.sw.ru) by relay.sw.ru with esmtp (Exim 4.89) (envelope-from ) id 1euKTR-0006SH-Dv; Fri, 09 Mar 2018 19:02:25 +0300 From: Vladimir Sementsov-Ogievskiy To: qemu-devel@nongnu.org, qemu-block@nongnu.org Date: Fri, 9 Mar 2018 19:02:23 +0300 Message-Id: <20180309160224.78821-2-vsementsov@virtuozzo.com> X-Mailer: git-send-email 2.11.1 In-Reply-To: <20180309160224.78821-1-vsementsov@virtuozzo.com> References: <20180309160224.78821-1-vsementsov@virtuozzo.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [fuzzy] X-Received-From: 185.231.240.75 Subject: [Qemu-devel] [PATCH v5 1/2] block/accounting: introduce latency histogram X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, armbru@redhat.com, mreitz@redhat.com, nshirokovskiy@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Introduce latency histogram statics for block devices. For each accounted operation type latency region [0, +inf) is divided into subregions by several points. Then, calculate hits for each subregion. Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/block/accounting.h | 35 ++++++++++++++++++ block/accounting.c | 91 ++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 126 insertions(+) diff --git a/include/block/accounting.h b/include/block/accounting.h index b833d26d6c..6eb2dd9ed3 100644 --- a/include/block/accounting.h +++ b/include/block/accounting.h @@ -27,6 +27,7 @@ #include "qemu/timed-average.h" #include "qemu/thread.h" +#include "qapi-types.h" typedef struct BlockAcctTimedStats BlockAcctTimedStats; typedef struct BlockAcctStats BlockAcctStats; @@ -45,6 +46,36 @@ struct BlockAcctTimedStats { QSLIST_ENTRY(BlockAcctTimedStats) entries; }; +typedef struct BlockLatencyHistogram { + /* The following histogram is represented like this: + * + * 5| * + * 4| * + * 3| * * + * 2| * * * + * 1| * * * * + * +------------------ + * 10 50 100 + * + * BlockLatencyHistogram histogram = { + * .nbins = 4, + * .boundaries = {10, 50, 100}, + * .bins = {3, 1, 5, 2}, + * }; + * + * @boundaries array define histogram intervals as follows: + * [0, boundaries[0]), [boundaries[0], boundaries[1]), ... + * [boundaries[nbins-2], +inf) + * + * So, for example above, histogram intervals are: + * [0, 10), [10, 50), [50, 100), [100, +inf) + */ + int nbins; + uint64_t *boundaries; /* @nbins-1 numbers here + (all boundaries, except 0 and +inf) */ + uint64_t *bins; +} BlockLatencyHistogram; + struct BlockAcctStats { QemuMutex lock; uint64_t nr_bytes[BLOCK_MAX_IOTYPE]; @@ -57,6 +88,7 @@ struct BlockAcctStats { QSLIST_HEAD(, BlockAcctTimedStats) intervals; bool account_invalid; bool account_failed; + BlockLatencyHistogram latency_histogram[BLOCK_MAX_IOTYPE]; }; typedef struct BlockAcctCookie { @@ -82,5 +114,8 @@ void block_acct_merge_done(BlockAcctStats *stats, enum BlockAcctType type, int64_t block_acct_idle_time_ns(BlockAcctStats *stats); double block_acct_queue_depth(BlockAcctTimedStats *stats, enum BlockAcctType type); +int block_latency_histogram_set(BlockAcctStats *stats, enum BlockAcctType type, + uint64List *boundaries); +void block_latency_histograms_clear(BlockAcctStats *stats); #endif diff --git a/block/accounting.c b/block/accounting.c index 87ef5bbfaa..70a3d9a426 100644 --- a/block/accounting.c +++ b/block/accounting.c @@ -94,6 +94,94 @@ void block_acct_start(BlockAcctStats *stats, BlockAcctCookie *cookie, cookie->type = type; } +/* block_latency_histogram_compare_func: + * Compare @key with interval [@it[0], @it[1]). + * Return: -1 if @key < @it[0] + * 0 if @key in [@it[0], @it[1]) + * +1 if @key >= @it[1] + */ +static int block_latency_histogram_compare_func(const void *key, const void *it) +{ + uint64_t k = *(uint64_t *)key; + uint64_t a = ((uint64_t *)it)[0]; + uint64_t b = ((uint64_t *)it)[1]; + + return k < a ? -1 : (k < b ? 0 : 1); +} + +static void block_latency_histogram_account(BlockLatencyHistogram *hist, + int64_t latency_ns) +{ + uint64_t *pos; + + if (hist->bins == NULL) { + /* histogram disabled */ + return; + } + + + if (latency_ns < hist->boundaries[0]) { + hist->bins[0]++; + return; + } + + if (latency_ns >= hist->boundaries[hist->nbins - 2]) { + hist->bins[hist->nbins - 1]++; + return; + } + + pos = bsearch(&latency_ns, hist->boundaries, hist->nbins - 2, + sizeof(hist->boundaries[0]), + block_latency_histogram_compare_func); + assert(pos != NULL); + + hist->bins[pos - hist->boundaries + 1]++; +} + +int block_latency_histogram_set(BlockAcctStats *stats, enum BlockAcctType type, + uint64List *boundaries) +{ + BlockLatencyHistogram *hist = &stats->latency_histogram[type]; + uint64List *entry; + uint64_t *ptr; + uint64_t prev = 0; + int new_nbins = 1; + + for (entry = boundaries; entry; entry = entry->next) { + if (entry->value <= prev) { + return -EINVAL; + } + new_nbins++; + prev = entry->value; + } + + hist->nbins = new_nbins; + g_free(hist->boundaries); + hist->boundaries = g_new(uint64_t, hist->nbins - 1); + for (entry = boundaries, ptr = hist->boundaries; entry; + entry = entry->next, ptr++) + { + *ptr = entry->value; + } + + g_free(hist->bins); + hist->bins = g_new0(uint64_t, hist->nbins); + + return 0; +} + +void block_latency_histograms_clear(BlockAcctStats *stats) +{ + int i; + + for (i = 0; i < BLOCK_MAX_IOTYPE; i++) { + BlockLatencyHistogram *hist = &stats->latency_histogram[i]; + g_free(hist->bins); + g_free(hist->boundaries); + memset(hist, 0, sizeof(*hist)); + } +} + static void block_account_one_io(BlockAcctStats *stats, BlockAcctCookie *cookie, bool failed) { @@ -116,6 +204,9 @@ static void block_account_one_io(BlockAcctStats *stats, BlockAcctCookie *cookie, stats->nr_ops[cookie->type]++; } + block_latency_histogram_account(&stats->latency_histogram[cookie->type], + latency_ns); + if (!failed || stats->account_failed) { stats->total_time_ns[cookie->type] += latency_ns; stats->last_access_time_ns = time_ns; From patchwork Fri Mar 9 16:02:24 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vladimir Sementsov-Ogievskiy X-Patchwork-Id: 883729 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=nongnu.org (client-ip=2001:4830:134:3::11; helo=lists.gnu.org; envelope-from=qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=virtuozzo.com Received: from lists.gnu.org (lists.gnu.org [IPv6:2001:4830:134:3::11]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3zyXGr0HPXz9sbv for ; Sat, 10 Mar 2018 03:03:14 +1100 (AEDT) Received: from localhost ([::1]:46156 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euKUB-0006ZD-O7 for incoming@patchwork.ozlabs.org; Fri, 09 Mar 2018 11:03:11 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56648) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1euKTZ-0006Yv-Lu for qemu-devel@nongnu.org; Fri, 09 Mar 2018 11:02:36 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1euKTU-0005yy-UF for qemu-devel@nongnu.org; Fri, 09 Mar 2018 11:02:33 -0500 Received: from relay.sw.ru ([185.231.240.75]:40042) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1euKTU-0005xw-IL; Fri, 09 Mar 2018 11:02:28 -0500 Received: from msk-vpn.virtuozzo.com ([195.214.232.6] helo=kvm.sw.ru) by relay.sw.ru with esmtp (Exim 4.89) (envelope-from ) id 1euKTR-0006SH-Ns; Fri, 09 Mar 2018 19:02:25 +0300 From: Vladimir Sementsov-Ogievskiy To: qemu-devel@nongnu.org, qemu-block@nongnu.org Date: Fri, 9 Mar 2018 19:02:24 +0300 Message-Id: <20180309160224.78821-3-vsementsov@virtuozzo.com> X-Mailer: git-send-email 2.11.1 In-Reply-To: <20180309160224.78821-1-vsementsov@virtuozzo.com> References: <20180309160224.78821-1-vsementsov@virtuozzo.com> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [fuzzy] X-Received-From: 185.231.240.75 Subject: [Qemu-devel] [PATCH v5 2/2] qapi: add block latency histogram interface X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, vsementsov@virtuozzo.com, armbru@redhat.com, mreitz@redhat.com, nshirokovskiy@virtuozzo.com, den@openvz.org Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" Set (and clear) histogram through new command block-latency-histogram-set and show new statistics in query-blockstats results. Signed-off-by: Vladimir Sementsov-Ogievskiy --- qapi/block-core.json | 111 ++++++++++++++++++++++++++++++++++++++++++++++++++- block/qapi.c | 41 +++++++++++++++++++ blockdev.c | 43 ++++++++++++++++++++ 3 files changed, 194 insertions(+), 1 deletion(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index 5c5921bfb7..b4dc9af45a 100644 --- a/qapi/block-core.json +++ b/qapi/block-core.json @@ -451,6 +451,106 @@ 'status': 'DirtyBitmapStatus'} } ## +# @XBlockLatencyHistogramInfo: +# +# Block latency histogram. +# +# @boundaries: list of interval boundary values in nanoseconds, all greater +# than zero and in ascending order. +# For example, the list [10, 50, 100] produces the following +# histogram intervals: [0, 10), [10, 50), [50, 100), [100, +inf). +# +# @bins: list of io request counts corresponding to histogram intervals. +# len(@bins) = len(@boundaries) + 1 +# For the example above, @bins may be something like [3, 1, 5, 2], +# and corresponding histogram looks like: +# +# 5| * +# 4| * +# 3| * * +# 2| * * * +# 1| * * * * +# +------------------ +# 10 50 100 +# +# Since: 2.12 +## +{ 'struct': 'XBlockLatencyHistogramInfo', + 'data': {'boundaries': ['uint64'], 'bins': ['uint64'] } } + +## +# @x-block-latency-histogram-set: +# +# Manage read, write and flush latency histograms for the device. +# +# If only @device parameter is specified, remove all present latency histograms +# for the device. Otherwise, add/reset some of (or all) latency histograms. +# +# @device: device name to set latency histogram for. +# +# @boundaries: list of interval boundary values (see description in +# XBlockLatencyHistogramInfo definition). If specified, all +# latency histograms are removed, and empty ones created for all +# io types with intervals corresponding to @boundaries (except for +# io types, for which specific boundaries are set through the +# following parameters). +# +# @boundaries-read: list of interval boundary values for read latency +# histogram. If specified, old read latency histogram is +# removed, and empty one created with interavals +# corresponding to @boundaries-read. The parameter has higher +# priority then @boundaries. +# +# @boundaries-write: list of interaval boundary values for write latency +# histogram. +# +# @boundaries-flush: list of interaval boundary values for flush latency +# histogram. +# +# Returns: error if device is not found or @boundaries* arrays are invalid. +# +# Since: 2.12 +# +# Example: set new histograms for all io types with intervals +# [0, 10), [10, 50), [50, 100), [100, +inf): +# +# -> { "execute": "block-latency-histogram-set", +# "arguments": { "device": "drive0", +# "boundaries": [10, 50, 100] } } +# <- { "return": {} } +# +# Example: set new histogram only for write, other histograms will remain +# not changed (or not created): +# +# -> { "execute": "block-latency-histogram-set", +# "arguments": { "device": "drive0", +# "boundaries-write": [10, 50, 100] } } +# <- { "return": {} } +# +# Example: set new histograms with the following intervals: +# read, flush: [0, 10), [10, 50), [50, 100), [100, +inf) +# write: [0, 1000), [1000, 5000), [5000, +inf) +# +# -> { "execute": "block-latency-histogram-set", +# "arguments": { "device": "drive0", +# "boundaries": [10, 50, 100], +# "boundaries-write": [1000, 5000] } } +# <- { "return": {} } +# +# Example: remove all latency histograms: +# +# -> { "execute": "block-latency-histogram-set", +# "arguments": { "device": "drive0" } } +# <- { "return": {} } +## +{ 'command': 'x-block-latency-histogram-set', + 'data': {'device': 'str', + '*boundaries': ['uint64'], + '*boundaries-read': ['uint64'], + '*boundaries-write': ['uint64'], + '*boundaries-flush': ['uint64'] } } + +## # @BlockInfo: # # Block device information. This structure describes a virtual device and @@ -730,6 +830,12 @@ # @timed_stats: Statistics specific to the set of previously defined # intervals of time (Since 2.5) # +# @x_rd_latency_histogram: @BlockLatencyHistogramInfo. (Since 2.12) +# +# @x_wr_latency_histogram: @BlockLatencyHistogramInfo. (Since 2.12) +# +# @x_flush_latency_histogram: @BlockLatencyHistogramInfo. (Since 2.12) +# # Since: 0.14.0 ## { 'struct': 'BlockDeviceStats', @@ -742,7 +848,10 @@ 'failed_flush_operations': 'int', 'invalid_rd_operations': 'int', 'invalid_wr_operations': 'int', 'invalid_flush_operations': 'int', 'account_invalid': 'bool', 'account_failed': 'bool', - 'timed_stats': ['BlockDeviceTimedStats'] } } + 'timed_stats': ['BlockDeviceTimedStats'], + '*x_rd_latency_histogram': 'XBlockLatencyHistogramInfo', + '*x_wr_latency_histogram': 'XBlockLatencyHistogramInfo', + '*x_flush_latency_histogram': 'XBlockLatencyHistogramInfo' } } ## # @BlockStats: diff --git a/block/qapi.c b/block/qapi.c index 1fdeb1ef2f..32bd0d45c4 100644 --- a/block/qapi.c +++ b/block/qapi.c @@ -394,6 +394,37 @@ static void bdrv_query_info(BlockBackend *blk, BlockInfo **p_info, qapi_free_BlockInfo(info); } +static uint64List *uint64_list(uint64_t *list, int size) +{ + int i; + uint64List *out_list = NULL; + uint64List **pout_list = &out_list; + + for (i = 0; i < size; i++) { + uint64List *entry = g_new(uint64List, 1); + entry->value = list[i]; + *pout_list = entry; + pout_list = &entry->next; + } + + *pout_list = NULL; + + return out_list; +} + +static void bdrv_latency_histogram_stats(BlockLatencyHistogram *hist, + bool *not_null, + XBlockLatencyHistogramInfo **info) +{ + *not_null = hist->bins != NULL; + if (*not_null) { + *info = g_new0(XBlockLatencyHistogramInfo, 1); + + (*info)->boundaries = uint64_list(hist->boundaries, hist->nbins - 1); + (*info)->bins = uint64_list(hist->bins, hist->nbins); + } +} + static void bdrv_query_blk_stats(BlockDeviceStats *ds, BlockBackend *blk) { BlockAcctStats *stats = blk_get_stats(blk); @@ -459,6 +490,16 @@ static void bdrv_query_blk_stats(BlockDeviceStats *ds, BlockBackend *blk) dev_stats->avg_wr_queue_depth = block_acct_queue_depth(ts, BLOCK_ACCT_WRITE); } + + bdrv_latency_histogram_stats(&stats->latency_histogram[BLOCK_ACCT_READ], + &ds->has_x_rd_latency_histogram, + &ds->x_rd_latency_histogram); + bdrv_latency_histogram_stats(&stats->latency_histogram[BLOCK_ACCT_WRITE], + &ds->has_x_wr_latency_histogram, + &ds->x_wr_latency_histogram); + bdrv_latency_histogram_stats(&stats->latency_histogram[BLOCK_ACCT_FLUSH], + &ds->has_x_flush_latency_histogram, + &ds->x_flush_latency_histogram); } static BlockStats *bdrv_query_bds_stats(BlockDriverState *bs, diff --git a/blockdev.c b/blockdev.c index 3fb1ca803c..8842d0aea2 100644 --- a/blockdev.c +++ b/blockdev.c @@ -4179,6 +4179,49 @@ void qmp_x_blockdev_set_iothread(const char *node_name, StrOrNull *iothread, aio_context_release(old_context); } +void qmp_x_block_latency_histogram_set( + const char *device, + bool has_boundaries, uint64List *boundaries, + bool has_boundaries_read, uint64List *boundaries_read, + bool has_boundaries_write, uint64List *boundaries_write, + bool has_boundaries_flush, uint64List *boundaries_flush, + Error **errp) +{ + BlockBackend *blk = blk_by_name(device); + BlockAcctStats *stats; + + if (!blk) { + error_setg(errp, "Device '%s' not found", device); + return; + } + stats = blk_get_stats(blk); + + if (!has_boundaries && !has_boundaries_read && !has_boundaries_write && + !has_boundaries_flush) + { + block_latency_histograms_clear(stats); + return; + } + + if (has_boundaries || has_boundaries_read) { + block_latency_histogram_set( + stats, BLOCK_ACCT_READ, + has_boundaries_read ? boundaries_read : boundaries); + } + + if (has_boundaries || has_boundaries_write) { + block_latency_histogram_set( + stats, BLOCK_ACCT_WRITE, + has_boundaries_write ? boundaries_write : boundaries); + } + + if (has_boundaries || has_boundaries_flush) { + block_latency_histogram_set( + stats, BLOCK_ACCT_FLUSH, + has_boundaries_flush ? boundaries_flush : boundaries); + } +} + QemuOptsList qemu_common_drive_opts = { .name = "drive", .head = QTAILQ_HEAD_INITIALIZER(qemu_common_drive_opts.head),