From patchwork Mon Dec 18 22:51:37 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Michelson X-Patchwork-Id: 850487 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3z0xB62qdfz9t2M for ; Tue, 19 Dec 2017 09:52:14 +1100 (AEDT) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 52311BF2; Mon, 18 Dec 2017 22:51:45 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 47AC8BE0 for ; Mon, 18 Dec 2017 22:51:42 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 55017FE for ; Mon, 18 Dec 2017 22:51:41 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D8DD937EE9 for ; Mon, 18 Dec 2017 22:51:40 +0000 (UTC) Received: from monae.redhat.com (ovpn-123-164.rdu2.redhat.com [10.10.123.164]) by smtp.corp.redhat.com (Postfix) with ESMTP id 76AB917AE0 for ; Mon, 18 Dec 2017 22:51:40 +0000 (UTC) From: Mark Michelson To: dev@openvswitch.org Date: Mon, 18 Dec 2017 16:51:37 -0600 Message-Id: <20171218225139.13789-2-mmichels@redhat.com> In-Reply-To: <20171218225139.13789-1-mmichels@redhat.com> References: <20171218225139.13789-1-mmichels@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Mon, 18 Dec 2017 22:51:40 +0000 (UTC) X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH 1/3] Add performance measuring API X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org This is similar to the existing coverage and perf-counter APIs in OVS. However, rather than keeping counters, this is aimed at timing how long operations take to perform. "Operations" in this case can be anything from a loop iteration, to a function, to something more complex. The library will keep track of how long it takes to perform the particular operations and will maintain statistics of those running times. Statistics for a particular operation can be queried from the command line by using ovs-appctl -t performance/show . The API is designed to be pretty small. The expected steps are as follows: 1) Create a performance measurement, providing a unique name, using performance_create() 2) Add calls to start_sample() and end_sample() to mark the start and stop of the operation you wish to measure. 3) Periodically call performance_run() in order to compile statistics. 4) Upon completion (likely program shutdown), call performance_destroy() to clean up. Signed-off-by: Mark Michelson --- lib/automake.mk | 2 + lib/performance.c | 335 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ lib/performance.h | 41 +++++++ 3 files changed, 378 insertions(+) create mode 100644 lib/performance.c create mode 100644 lib/performance.h diff --git a/lib/automake.mk b/lib/automake.mk index effe5b5c2..a3ac7bc1d 100644 --- a/lib/automake.mk +++ b/lib/automake.mk @@ -205,6 +205,8 @@ lib_libopenvswitch_la_SOURCES = \ lib/pcap-file.h \ lib/perf-counter.h \ lib/perf-counter.c \ + lib/performance.h \ + lib/performance.c \ lib/poll-loop.c \ lib/process.c \ lib/process.h \ diff --git a/lib/performance.c b/lib/performance.c new file mode 100644 index 000000000..0d0de3645 --- /dev/null +++ b/lib/performance.c @@ -0,0 +1,335 @@ +/* Copyright (c) 2017 Red Hat, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#include + +#include "performance.h" +#include "timer.h" +#include "timeval.h" +#include "openvswitch/shash.h" +#include "openvswitch/vlog.h" +#include "unixctl.h" +#include "openvswitch/dynamic-string.h" +#include "openvswitch/poll-loop.h" + +VLOG_DEFINE_THIS_MODULE(performance); + +struct sample { + /* Time when we started this sample */ + long long int start_time; + /* Time when we ended this sample */ + long long int end_time; + /* Amount of time elapsed during timing. + * Equal to end_time - start_time + */ + long long int elapsed; +}; + +struct sample_vec { + struct sample *samples; + size_t n_samples; + size_t capacity; +}; + +static void +add_sample(struct sample_vec *vec, struct sample *new_sample) +{ + if (vec->capacity == vec->n_samples) { + vec->samples = x2nrealloc(vec->samples, &vec->capacity, + sizeof *vec->samples); + } + + vec->samples[vec->n_samples++] = *new_sample; +} + +static int +find_earliest(const struct sample_vec *vec, long long int age_ms) +{ + long long int cutoff = time_msec() - age_ms; + + for (size_t i = 0; i < vec->n_samples; i++) { + if (vec->samples[i].end_time >= cutoff) { + return i; + } + } + + /* Either the vector is empty or all times are + * older than the cutoff. + */ + return -1; +} + +static long long int +average(const struct sample *samples, size_t num_samples) +{ + /* Avoid division by zero */ + if (num_samples == 0) { + return 0; + } + + long long int sum = 0; + for (size_t i = 0; i < num_samples; i++) { + sum += samples[i].elapsed; + } + + return sum / num_samples; +} + +static long long int +percentile(const struct sample *samples, size_t num_samples, int percentile) +{ + if (num_samples == 0) { + return 0; + } + + size_t pctl = num_samples * percentile / 100; + return samples[pctl].elapsed; +} + +static void +cull_old_times(struct sample_vec *vec, long long int age_ms) +{ + int i = find_earliest(vec, age_ms); + + if (i <= 0) { + return; + } + + size_t new_size = vec->n_samples - i; + memmove(vec->samples, &vec->samples[i], new_size * sizeof *vec->samples); + vec->n_samples = new_size; +} + +struct stats { + long long int min; + long long int max; + long long int average; + long long int percentile; + long long int num_samples; +}; + +struct performance { + struct sample_vec vec; + long long int sample_rate; + struct timer timer; + struct sample cur_sample; + struct stats one_min; + struct stats five_min; + struct stats ten_min; + struct ovsdb_idl *idl; +}; + +struct shash performances = SHASH_INITIALIZER(&performances); + +static void +format_stats(struct ds *s, const char *prefix, struct stats *stats) +{ + if (stats->num_samples) { + ds_put_format(s, "\t%s samples: %lld\n", prefix, stats->num_samples); + ds_put_format(s, "\t%s minimum: %lld\n", prefix, stats->min); + ds_put_format(s, "\t%s maximum: %lld\n", prefix, stats->max); + ds_put_format(s, "\t%s average: %lld\n", prefix, stats->average); + ds_put_format(s, "\t%s 95th percentile: %lld\n\n", prefix, + stats->percentile); + } else { + ds_put_format(s, "\t%s samples: 0\n", prefix); + ds_put_format(s, "\t%s minimum: N/A\n", prefix); + ds_put_format(s, "\t%s maximum: N/A\n", prefix); + ds_put_format(s, "\t%s average: N/A\n", prefix); + ds_put_format(s, "\t%s 95th percentile: N/A\n\n", prefix); + } +} + +static void +performance_show(struct unixctl_conn *conn, int argc OVS_UNUSED, + const char *argv[], void *ignore OVS_UNUSED) +{ + struct performance *perf = shash_find_data(&performances, argv[1]); + + if (!perf) { + unixctl_command_reply_error(conn, "No such performance"); + return; + } + + struct ds s = DS_EMPTY_INITIALIZER; + ds_put_format(&s, "Statistics for '%s'\n", argv[1]); + + format_stats(&s, "1 minute", &perf->one_min); + format_stats(&s, "5 minute", &perf->five_min); + format_stats(&s, "10 minute", &perf->ten_min); + + unixctl_command_reply(conn, ds_cstr(&s)); + ds_destroy(&s); +} + +static void +do_init_performance(void) +{ + unixctl_command_register("performance/show", "", 1, 1, + performance_show, NULL); +} + +static void +performance_init(void) +{ + static pthread_once_t once = PTHREAD_ONCE_INIT; + pthread_once(&once, do_init_performance); +} + +void +performance_create(const char *name, long long int sample_rate) +{ + performance_init(); + + struct performance *perf = xcalloc(1, sizeof *perf); + perf->sample_rate = sample_rate; + timer_set_duration(&perf->timer, perf->sample_rate); + shash_add(&performances, name, perf); +} + +void +performance_destroy(const char *name) +{ + struct performance *perf = shash_find_and_delete(&performances, name); + if (perf) { + free(perf->vec.samples); + } + + free(perf); +} + +void +performance_set_sample_rate(const char *name, long long int sample_rate) +{ + struct performance *perf = shash_find_data(&performances, name); + if (!perf || perf->sample_rate == sample_rate) { + return; + } + perf->sample_rate = sample_rate; + /* If we're shortening the sample rate, then make the timer + * expire more quickly. Otherwise, leave it alone. + */ + if (timer_msecs_until_expired(&perf->timer) > sample_rate) { + timer_set_duration(&perf->timer, sample_rate); + } +} + +bool +start_sample(const char *name) +{ + struct performance *perf = shash_find_data(&performances, name); + if (!perf) { + return false; + } + + /* We already started sampling. Need an end before + * we start another sample + */ + if (perf->cur_sample.start_time) { + return false; + } + + perf->cur_sample.start_time = time_msec(); + return true; +} + +bool +end_sample(const char *name) +{ + struct performance *perf = shash_find_data(&performances, name); + if (!perf) { + return false; + } + + /* We can't end a sample if we haven't started one */ + if (!perf->cur_sample.start_time) { + return false; + } + + perf->cur_sample.end_time = time_msec(); + perf->cur_sample.elapsed = perf->cur_sample.end_time + - perf->cur_sample.start_time; + + add_sample(&perf->vec, &perf->cur_sample); + + memset(&perf->cur_sample, 0, sizeof perf->cur_sample); + return true; +} + +static int +cmp_times(const void *left_, const void *right_) +{ + const struct sample *left = left_; + const struct sample *right = right_; + + return left->elapsed == right->elapsed ? 0 + : left->elapsed > right->elapsed; +} + +static struct sample * +sort_times(const struct sample *by_time, size_t vec_size) +{ + struct sample *copy = xmalloc(vec_size * sizeof *copy); + memcpy(copy, by_time, vec_size * sizeof *copy); + qsort(copy, vec_size, sizeof *copy, cmp_times); + + return copy; +} + +static int +get_stats(const struct sample_vec *vec, long long int age_ms, + struct stats *stats) +{ + int start_idx = find_earliest(vec, age_ms); + if (start_idx < 0) { + memset(stats, 0, sizeof *stats); + return -1; + } + size_t vec_size = vec->n_samples - start_idx; + + struct sample *by_time = &vec->samples[start_idx]; + struct sample *by_elapsed = sort_times(by_time, vec_size); + + stats->min = by_elapsed[0].elapsed; + stats->max = by_elapsed[vec_size - 1].elapsed; + stats->average = average(by_time, vec_size); + stats->percentile = percentile(by_elapsed, vec_size, 95); + stats->num_samples = vec_size; + + free(by_elapsed); + + return 0; +} + +void +performance_run(const char *name) +{ + struct performance *perf = shash_find_data(&performances, name); + if (!perf) { + return; + } + if (!timer_expired(&perf->timer)) { + timer_wait(&perf->timer); + return; + } + + get_stats(&perf->vec, 60000, &perf->one_min); + get_stats(&perf->vec, 300000, &perf->five_min); + get_stats(&perf->vec, 600000, &perf->ten_min); + + cull_old_times(&perf->vec, 600000); + timer_set_duration(&perf->timer, perf->sample_rate); + timer_wait(&perf->timer); +} diff --git a/lib/performance.h b/lib/performance.h new file mode 100644 index 000000000..64d460708 --- /dev/null +++ b/lib/performance.h @@ -0,0 +1,41 @@ +/* Copyright (c) 2017 Red Hat, Inc. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at: + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +#ifndef PERFORMANCE_H +#define PERFORMANCE_H 1 + +#include + +/* This file provides a method for timing operations in OVS. + * + * The expected operation is as follows: + * 1) Call performance_create(), supplying a unique name for the + * operation that is being measured. + * 2) Place calls to start_sample() and end_sample() at the beginning + * and end of the operation you wish to measure. + * 3) Periodically call performance_run() so that statistics can be + * gathered based on the recorded times. If you are running a loop, + * calling this once per loop is a good practice. + * 4) When complete, call performance_destroy() to clean up. + */ + +void performance_create(const char *name, long long int sample_rate); +void performance_destroy(const char *name); +void performance_set_sample_rate(const char * name, long long int sample_rate); +bool start_sample(const char *name); +bool end_sample(const char *name); +void performance_run(const char *name); + +#endif /* performance.h */ From patchwork Mon Dec 18 22:51:38 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Michelson X-Patchwork-Id: 850488 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3z0xBn2vJFz9s4s for ; Tue, 19 Dec 2017 09:52:49 +1100 (AEDT) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 41ADAC0C; Mon, 18 Dec 2017 22:51:46 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id 684B1BE1 for ; Mon, 18 Dec 2017 22:51:42 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id B468B4F8 for ; Mon, 18 Dec 2017 22:51:41 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 597CB37E80 for ; Mon, 18 Dec 2017 22:51:41 +0000 (UTC) Received: from monae.redhat.com (ovpn-123-164.rdu2.redhat.com [10.10.123.164]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0D55917AE0 for ; Mon, 18 Dec 2017 22:51:40 +0000 (UTC) From: Mark Michelson To: dev@openvswitch.org Date: Mon, 18 Dec 2017 16:51:38 -0600 Message-Id: <20171218225139.13789-3-mmichels@redhat.com> In-Reply-To: <20171218225139.13789-1-mmichels@redhat.com> References: <20171218225139.13789-1-mmichels@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.29]); Mon, 18 Dec 2017 22:51:41 +0000 (UTC) X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH 2/3] Save performance stats to database. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org This commit builds on the performance library by storing compiled statistics in the OVS vswitchd database. This can make it more useful for administrators to keep tabs on performance in real time and make adjustments if necessary. Signed-off-by: Mark Michelson --- lib/performance.c | 69 ++++++++++++++++++++++++++++++++++++++++++++-- lib/performance.h | 8 ++++-- vswitchd/vswitch.ovsschema | 39 ++++++++++++++++++++++++-- vswitchd/vswitch.xml | 69 ++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 179 insertions(+), 6 deletions(-) diff --git a/lib/performance.c b/lib/performance.c index 0d0de3645..b71901d78 100644 --- a/lib/performance.c +++ b/lib/performance.c @@ -23,6 +23,7 @@ #include "unixctl.h" #include "openvswitch/dynamic-string.h" #include "openvswitch/poll-loop.h" +#include "vswitch-idl.h" VLOG_DEFINE_THIS_MODULE(performance); @@ -189,12 +190,14 @@ performance_init(void) } void -performance_create(const char *name, long long int sample_rate) +performance_create(const char *name, long long int sample_rate, + struct ovsdb_idl *idl) { performance_init(); struct performance *perf = xcalloc(1, sizeof *perf); perf->sample_rate = sample_rate; + perf->idl = idl; timer_set_duration(&perf->timer, perf->sample_rate); shash_add(&performances, name, perf); } @@ -207,6 +210,15 @@ performance_destroy(const char *name) free(perf->vec.samples); } + const struct ovsrec_performance *performance; + const struct ovsrec_performance *next; + + OVSREC_PERFORMANCE_FOR_EACH_SAFE (performance, next, perf->idl) { + if (!strcmp(performance->name, name)) { + ovsrec_performance_delete(performance); + } + } + free(perf); } @@ -313,8 +325,59 @@ get_stats(const struct sample_vec *vec, long long int age_ms, return 0; } +static void +save_stats_to_db(const struct performance *perf, const char *name, + struct ovsdb_idl_txn *txn) +{ + if (!txn) { + return; + } + + const struct ovsrec_performance *performance; + + OVSREC_PERFORMANCE_FOR_EACH (performance, perf->idl) { + if (!strcmp(performance->name, name)) { + break; + } + } + + if (!performance) { + struct ovsrec_performance *new_performance; + new_performance = ovsrec_performance_insert(txn); + ovsrec_performance_set_name(new_performance, name); + performance = new_performance; + } + + ovsrec_performance_set_one_minute_samples(performance, + perf->one_min.num_samples); + ovsrec_performance_set_one_minute_max(performance, perf->one_min.max); + ovsrec_performance_set_one_minute_min(performance, perf->one_min.min); + ovsrec_performance_set_one_minute_average(performance, + perf->one_min.average); + ovsrec_performance_set_one_minute_ninety_five_percentile(performance, + perf->one_min.percentile); + + ovsrec_performance_set_five_minute_samples(performance, + perf->five_min.num_samples); + ovsrec_performance_set_five_minute_max(performance, perf->five_min.max); + ovsrec_performance_set_five_minute_min(performance, perf->five_min.min); + ovsrec_performance_set_five_minute_average(performance, + perf->five_min.average); + ovsrec_performance_set_five_minute_ninety_five_percentile(performance, + perf->five_min.percentile); + + ovsrec_performance_set_ten_minute_samples(performance, + perf->ten_min.num_samples); + ovsrec_performance_set_ten_minute_max(performance, perf->ten_min.max); + ovsrec_performance_set_ten_minute_min(performance, perf->ten_min.min); + ovsrec_performance_set_ten_minute_average(performance, + perf->ten_min.average); + ovsrec_performance_set_ten_minute_ninety_five_percentile(performance, + perf->ten_min.percentile); +} + void -performance_run(const char *name) +performance_run(const char *name, struct ovsdb_idl_txn *txn) { struct performance *perf = shash_find_data(&performances, name); if (!perf) { @@ -329,6 +392,8 @@ performance_run(const char *name) get_stats(&perf->vec, 300000, &perf->five_min); get_stats(&perf->vec, 600000, &perf->ten_min); + save_stats_to_db(perf, name, txn); + cull_old_times(&perf->vec, 600000); timer_set_duration(&perf->timer, perf->sample_rate); timer_wait(&perf->timer); diff --git a/lib/performance.h b/lib/performance.h index 64d460708..3d86bf7de 100644 --- a/lib/performance.h +++ b/lib/performance.h @@ -31,11 +31,15 @@ * 4) When complete, call performance_destroy() to clean up. */ -void performance_create(const char *name, long long int sample_rate); +struct ovsdb_idl; +struct ovsdb_idl_txn; + +void performance_create(const char *name, long long int sample_rate, + struct ovsdb_idl *idl); void performance_destroy(const char *name); void performance_set_sample_rate(const char * name, long long int sample_rate); bool start_sample(const char *name); bool end_sample(const char *name); -void performance_run(const char *name); +void performance_run(const char *name, struct ovsdb_idl_txn *txn); #endif /* performance.h */ diff --git a/vswitchd/vswitch.ovsschema b/vswitchd/vswitch.ovsschema index 90e50b626..579483fdf 100644 --- a/vswitchd/vswitch.ovsschema +++ b/vswitchd/vswitch.ovsschema @@ -1,6 +1,6 @@ {"name": "Open_vSwitch", "version": "7.15.1", - "cksum": "3682332033 23608", + "cksum": "1712487757 24640", "tables": { "Open_vSwitch": { "columns": { @@ -648,4 +648,39 @@ "value": {"type": "integer", "minInteger": 0, "maxInteger": 4095}, - "min": 0, "max": "unlimited"}}}}}} + "min": 0, "max": "unlimited"}}}}, + "Performance": { + "columns": { + "name": { + "type": "string"}, + "one_minute_samples": { + "type": "integer"}, + "one_minute_min": { + "type": "integer"}, + "one_minute_max": { + "type": "integer"}, + "one_minute_average": { + "type": "integer"}, + "one_minute_ninety_five_percentile": { + "type": "integer"}, + "five_minute_samples": { + "type": "integer"}, + "five_minute_min": { + "type": "integer"}, + "five_minute_max": { + "type": "integer"}, + "five_minute_average": { + "type": "integer"}, + "five_minute_ninety_five_percentile": { + "type": "integer"}, + "ten_minute_samples": { + "type": "integer"}, + "ten_minute_min": { + "type": "integer"}, + "ten_minute_max": { + "type": "integer"}, + "ten_minute_average": { + "type": "integer"}, + "ten_minute_ninety_five_percentile": { + "type": "integer"}}, + "isRoot": true}}} diff --git a/vswitchd/vswitch.xml b/vswitchd/vswitch.xml index 21ffaf541..045d9b049 100644 --- a/vswitchd/vswitch.xml +++ b/vswitchd/vswitch.xml @@ -5399,4 +5399,73 @@ ovs-vsctl add-port br0 p0 -- set Interface p0 type=patch options:peer=p1 \ id. + + +

+ Performance measurements for operations within OVS. +

+ +

+ OVS and OVN perform repeated operations during their lifetimes. Some of + these operations are timed and have their measurements saved. From those + saved measurements, statistics are written to the + table at regular intervals. +

+ +

+ All times are measured in milliseconds. +

+ + Performance identifier. Must be unique amongst other performance + measurements. + + + The number of performance measurements taken over the past minute. + + + The minimum performance measurement taken over the past minute. + + + The maximum performance measurement taken over the past minute. + + + The average performance measurement taken over the past minute. + + + The ninety-fifth percentile performance measurement taken over the past + minute. + + + The number of performance measurements taken over the past five minutes. + + + The minimum performance measurement taken over the past five minutes. + + + The maximum performance measurement taken over the past five minutes. + + + The average performance measurement taken over the past five minutes. + + + The ninety-fifth percentile performance measurement taken over the past + five minutes. + + + The number of performance measurements taken over the past ten minutes. + + + The minimum performance measurement taken over the past ten minutes. + + + The maximum performance measurement taken over the past ten minutes. + + + The average performance measurement taken over the past ten minutes. + + + The ninety-fifth percentile performance measurement taken over the past + ten minutes. + +
From patchwork Mon Dec 18 22:51:39 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Michelson X-Patchwork-Id: 850489 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (mailfrom) smtp.mailfrom=openvswitch.org (client-ip=140.211.169.12; helo=mail.linuxfoundation.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Received: from mail.linuxfoundation.org (mail.linuxfoundation.org [140.211.169.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 3z0xCP3BSgz9t2M for ; Tue, 19 Dec 2017 09:53:20 +1100 (AEDT) Received: from mail.linux-foundation.org (localhost [127.0.0.1]) by mail.linuxfoundation.org (Postfix) with ESMTP id 382D6C13; Mon, 18 Dec 2017 22:51:47 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@mail.linuxfoundation.org Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id C65B0BE0 for ; Mon, 18 Dec 2017 22:51:42 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 4985F4FA for ; Mon, 18 Dec 2017 22:51:42 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id D01C8C049E39 for ; Mon, 18 Dec 2017 22:51:41 +0000 (UTC) Received: from monae.redhat.com (ovpn-123-164.rdu2.redhat.com [10.10.123.164]) by smtp.corp.redhat.com (Postfix) with ESMTP id 82E0817C0E for ; Mon, 18 Dec 2017 22:51:41 +0000 (UTC) From: Mark Michelson To: dev@openvswitch.org Date: Mon, 18 Dec 2017 16:51:39 -0600 Message-Id: <20171218225139.13789-4-mmichels@redhat.com> In-Reply-To: <20171218225139.13789-1-mmichels@redhat.com> References: <20171218225139.13789-1-mmichels@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Mon, 18 Dec 2017 22:51:41 +0000 (UTC) X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on smtp1.linux-foundation.org Subject: [ovs-dev] [PATCH 3/3] Measure performance of ovn-controller loop. X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ovs-dev-bounces@openvswitch.org Errors-To: ovs-dev-bounces@openvswitch.org This modifies ovn-controller to measure the amount of time it takes to detect a change in the southbound database, generate the resulting flow table, and write the flow table down to vswitchd. The statistics can be queried using: ovs-appctl -t ovn-controller performance/show ovn-controller-loop Signed-off-by: Mark Michelson --- ovn/controller/ovn-controller.c | 53 +++++++++++++++++++++++++++++++++++++++++ vswitchd/vswitch.xml | 12 ++++++++++ 2 files changed, 65 insertions(+) diff --git a/ovn/controller/ovn-controller.c b/ovn/controller/ovn-controller.c index c286ccbca..4f1769cfc 100644 --- a/ovn/controller/ovn-controller.c +++ b/ovn/controller/ovn-controller.c @@ -56,6 +56,9 @@ #include "stream.h" #include "unixctl.h" #include "util.h" +#include "timeval.h" +#include "timer.h" +#include "performance.h" VLOG_DEFINE_THIS_MODULE(main); @@ -66,6 +69,9 @@ static unixctl_cb_func inject_pkt; #define DEFAULT_BRIDGE_NAME "br-int" #define DEFAULT_PROBE_INTERVAL_MSEC 5000 +#define CONTROLLER_LOOP_PERFORMANCE_NAME "ovn-controller-loop" +#define DEFAULT_CONTROLLER_LOOP_PERFORMANCE_RATE 10000 + static void update_probe_interval(struct controller_ctx *, const char *ovnsb_remote); static void parse_options(int argc, char *argv[]); @@ -524,6 +530,26 @@ ctrl_register_ovs_idl(struct ovsdb_idl *ovs_idl) ovsdb_idl_add_column(ovs_idl, &ovsrec_ssl_col_ca_cert); ovsdb_idl_add_column(ovs_idl, &ovsrec_ssl_col_certificate); ovsdb_idl_add_column(ovs_idl, &ovsrec_ssl_col_private_key); + ovsdb_idl_add_table(ovs_idl, &ovsrec_table_performance); + ovsdb_idl_add_column(ovs_idl, &ovsrec_performance_col_name); + ovsdb_idl_add_column(ovs_idl, &ovsrec_performance_col_one_minute_samples); + ovsdb_idl_add_column(ovs_idl, &ovsrec_performance_col_one_minute_min); + ovsdb_idl_add_column(ovs_idl, &ovsrec_performance_col_one_minute_max); + ovsdb_idl_add_column(ovs_idl, &ovsrec_performance_col_one_minute_average); + ovsdb_idl_add_column(ovs_idl, + &ovsrec_performance_col_one_minute_ninety_five_percentile); + ovsdb_idl_add_column(ovs_idl, &ovsrec_performance_col_five_minute_samples); + ovsdb_idl_add_column(ovs_idl, &ovsrec_performance_col_five_minute_min); + ovsdb_idl_add_column(ovs_idl, &ovsrec_performance_col_five_minute_max); + ovsdb_idl_add_column(ovs_idl, &ovsrec_performance_col_five_minute_average); + ovsdb_idl_add_column(ovs_idl, + &ovsrec_performance_col_five_minute_ninety_five_percentile); + ovsdb_idl_add_column(ovs_idl, &ovsrec_performance_col_ten_minute_samples); + ovsdb_idl_add_column(ovs_idl, &ovsrec_performance_col_ten_minute_min); + ovsdb_idl_add_column(ovs_idl, &ovsrec_performance_col_ten_minute_max); + ovsdb_idl_add_column(ovs_idl, &ovsrec_performance_col_ten_minute_average); + ovsdb_idl_add_column(ovs_idl, + &ovsrec_performance_col_ten_minute_ninety_five_percentile); chassis_register_ovs_idl(ovs_idl); encaps_register_ovs_idl(ovs_idl); binding_register_ovs_idl(ovs_idl); @@ -637,8 +663,12 @@ main(int argc, char *argv[]) unixctl_command_register("inject-pkt", "MICROFLOW", 1, 1, inject_pkt, &pending_pkt); + performance_create(CONTROLLER_LOOP_PERFORMANCE_NAME, + DEFAULT_CONTROLLER_LOOP_PERFORMANCE_RATE, + ovs_idl_loop.idl); /* Main loop. */ exiting = false; + unsigned int our_seqno = 0; while (!exiting) { /* Check OVN SB database. */ char *new_ovnsb_remote = get_ovnsb_remote(ovs_idl_loop.idl); @@ -657,6 +687,11 @@ main(int argc, char *argv[]) .ovnsb_idl_txn = ovsdb_idl_loop_run(&ovnsb_idl_loop), }; + if (our_seqno != ovsdb_idl_get_seqno(ctx.ovnsb_idl) && + start_sample(CONTROLLER_LOOP_PERFORMANCE_NAME)) { + our_seqno = ovsdb_idl_get_seqno(ctx.ovnsb_idl); + } + update_probe_interval(&ctx, ovnsb_remote); update_ssl_config(ctx.ovs_idl); @@ -726,6 +761,8 @@ main(int argc, char *argv[]) ofctrl_put(&flow_table, &pending_ct_zones, get_nb_cfg(ctx.ovnsb_idl)); + end_sample(CONTROLLER_LOOP_PERFORMANCE_NAME); + hmap_destroy(&flow_table); } if (ctx.ovnsb_idl_txn) { @@ -754,6 +791,7 @@ main(int argc, char *argv[]) expr_addr_sets_destroy(&addr_sets); shash_destroy(&addr_sets); + } /* If we haven't handled the pending packet insertion @@ -790,6 +828,20 @@ main(int argc, char *argv[]) ofctrl_wait(); pinctrl_wait(&ctx); } + + const struct ovsrec_open_vswitch *cfg; + cfg = ovsrec_open_vswitch_first(ctx.ovs_idl); + if (cfg) { + const char *sample_rate = smap_get(&cfg->external_ids, + "ovn-controller-loop-sample-rate"); + int sample_rate_ms; + if (sample_rate && sample_rate[0] + && str_to_int(sample_rate, 10, &sample_rate_ms)) { + performance_set_sample_rate(CONTROLLER_LOOP_PERFORMANCE_NAME, + sample_rate_ms); + } + } + performance_run(CONTROLLER_LOOP_PERFORMANCE_NAME, ctx.ovs_idl_txn); ovsdb_idl_loop_commit_and_wait(&ovnsb_idl_loop); if (ovsdb_idl_loop_commit_and_wait(&ovs_idl_loop) == 1) { @@ -837,6 +889,7 @@ main(int argc, char *argv[]) poll_block(); } + performance_destroy("ovn-controller-loop"); unixctl_server_destroy(unixctl); lflow_destroy(); ofctrl_destroy(); diff --git a/vswitchd/vswitch.xml b/vswitchd/vswitch.xml index 045d9b049..145ae4a99 100644 --- a/vswitchd/vswitch.xml +++ b/vswitchd/vswitch.xml @@ -5415,6 +5415,18 @@ ovs-vsctl add-port br0 p0 -- set Interface p0 type=patch options:peer=p1 \

All times are measured in milliseconds.

+ +

+ The following performance measurements are recorded in this database: +

+
    +
  • + ovn-controller-loop. This measures the amount of time that + it takes for ovn-controller to detect a change in the OVN + southbound database, generate the resulting flow table, and write the + table out for ovs-vswitchd to process. +
    • +
Performance identifier. Must be unique amongst other performance measurements.