From patchwork Thu Jun 17 17:42:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Gray X-Patchwork-Id: 1493777 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.138; helo=smtp1.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Kpuf8u3g; dkim-atps=neutral Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G5TtD12tKz9sCD for ; Fri, 18 Jun 2021 03:43:14 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id E6D34842F5; Thu, 17 Jun 2021 17:43:11 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id HpRo5gsOA52x; Thu, 17 Jun 2021 17:43:10 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp1.osuosl.org (Postfix) with ESMTPS id 2FEB783D87; Thu, 17 Jun 2021 17:43:09 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 0A484C000E; Thu, 17 Jun 2021 17:43:09 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp3.osuosl.org (smtp3.osuosl.org [IPv6:2605:bc80:3010::136]) by lists.linuxfoundation.org (Postfix) with ESMTP id DDF62C000B for ; Thu, 17 Jun 2021 17:43:07 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id B7A2F6071E for ; Thu, 17 Jun 2021 17:43:07 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp3.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id UL81TbfFoX9M for ; Thu, 17 Jun 2021 17:43:07 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by smtp3.osuosl.org (Postfix) with ESMTPS id 20DDE605D3 for ; Thu, 17 Jun 2021 17:43:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1623951785; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2GlxTQrf1Y+I9cMi27Jv/UZ90ys52cfZ2jNIpbo88QM=; b=Kpuf8u3g9H0lhadlOvJAcNGaoRAHYwNXhqQcdK1DMo60aHbrw6fzRV7aMtuqkk8EfjpHCV guoooqUdgzVz7pPK4hpOl/R77zpJbfxYi6tFk1EefkSqBfNMJh+hreabYGDkXi3nR8+Qn0 OEt7M6ZVtF5TN1OLFGgFzBnULaBZ0P8= Received: from mail-pg1-f197.google.com (mail-pg1-f197.google.com [209.85.215.197]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-518-Um6mJsDiObOf2inZDC3T1g-1; Thu, 17 Jun 2021 13:43:04 -0400 X-MC-Unique: Um6mJsDiObOf2inZDC3T1g-1 Received: by mail-pg1-f197.google.com with SMTP id d8-20020a6558880000b02902225e17c66bso4194152pgu.23 for ; Thu, 17 Jun 2021 10:43:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2GlxTQrf1Y+I9cMi27Jv/UZ90ys52cfZ2jNIpbo88QM=; b=TpnHS8lwRAapXYWLAaGrEYm252g7nBV6D3ElPdzjJ7Ww0zcXsKySKe4lSlqlOrnjMJ Rmh8xkoBapPG50S7tHHU+2WKFJtTLZOL3SZ7E5qnidydFLncJD0plHP2q8EnKHLqIcrB 3libyIKC0S4SfrggzqvIqz3t3FioApBPP8rHrnSBVq1fx+sbEZ9sB2GeGSJfLz/BN3nm eZLM2fWBHInZ6LEcUVlY8zy32c6jCTb03UaLhe9UwANTSFEIMWXHd2qoBf5JCy6klTGj uC0NslTh8u1DJWDZl72kzsqrsCKRy8HW1l4Wf+XJeekEBgxZScRJkJTB021ULaRNilip ZyGA== X-Gm-Message-State: AOAM532IJrm7MRLIE+Ll4DVgGiP8duUr2U1XhouaAPiHWNLvdQnXfyXQ 06j+JJppr7JYQikxDr1174YpdFrGJNyC7yzHzBB+Loc+uSPROmU5q4mPipWj6poaPrVAm1r30Z1 wotGdeTj1aFQHd9vgvlkpZven0TcbebCqzctDzgcX8fbIFQBl/c0rEBiWWiDXenCzMkt5 X-Received: by 2002:a17:90a:6548:: with SMTP id f8mr4495753pjs.106.1623951782923; Thu, 17 Jun 2021 10:43:02 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxZYiTzlNOfuEYTB13JyTMsMsQ+mcs3fc4iXFCQ/eYKnM2I/FdrtOLCINUnyDz/FTbgdWjTig== X-Received: by 2002:a17:90a:6548:: with SMTP id f8mr4495732pjs.106.1623951782695; Thu, 17 Jun 2021 10:43:02 -0700 (PDT) Received: from wsfd-netdev91.ntdv.lab.eng.bos.redhat.com (nat-pool-bos-t.redhat.com. [66.187.233.206]) by smtp.gmail.com with ESMTPSA id y34sm5687873pfa.181.2021.06.17.10.43.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Jun 2021 10:43:02 -0700 (PDT) From: Mark Gray To: dev@openvswitch.org Date: Thu, 17 Jun 2021 13:42:51 -0400 Message-Id: <20210617174253.3459548-2-mark.d.gray@redhat.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210617174253.3459548-1-mark.d.gray@redhat.com> References: <20210617174253.3459548-1-mark.d.gray@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=mark.d.gray@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH ovn v2 1/3] ovn-macros.at: fix typo X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Signed-off-by: Mark Gray --- tests/ovn-macros.at | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/ovn-macros.at b/tests/ovn-macros.at index cd02b6986cc2..c92c07720112 100644 --- a/tests/ovn-macros.at +++ b/tests/ovn-macros.at @@ -184,7 +184,7 @@ ovn_start_northd() { # ovn-sbctl and ovn-nbctl use them by default, and starts ovn-northd running # against them. # -# Normally this starts an active northd and a backup norhtd. The following +# Normally this starts an active northd and a backup northd. The following # options are accepted to adjust that: # --backup-northd=none Don't start a backup northd. # --backup-northd=paused Start the backup northd in the paused state. From patchwork Thu Jun 17 17:42:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Gray X-Patchwork-Id: 1493779 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=140.211.166.136; helo=smtp3.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=a791hgxx; dkim-atps=neutral Received: from smtp3.osuosl.org (smtp3.osuosl.org [140.211.166.136]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G5TtH26Mhz9sSn for ; Fri, 18 Jun 2021 03:43:19 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp3.osuosl.org (Postfix) with ESMTP id 48F8960AD9; Thu, 17 Jun 2021 17:43:17 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp3.osuosl.org ([127.0.0.1]) by localhost (smtp3.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NpYfKTr5XoUF; Thu, 17 Jun 2021 17:43:16 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp3.osuosl.org (Postfix) with ESMTPS id 796FC60ACE; Thu, 17 Jun 2021 17:43:15 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 85BD6C0027; Thu, 17 Jun 2021 17:43:13 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138]) by lists.linuxfoundation.org (Postfix) with ESMTP id 23F07C000B for ; Thu, 17 Jun 2021 17:43:12 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp1.osuosl.org (Postfix) with ESMTP id 0BFC083DC1 for ; Thu, 17 Jun 2021 17:43:12 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp1.osuosl.org (amavisd-new); dkim=pass (1024-bit key) header.d=redhat.com Received: from smtp1.osuosl.org ([127.0.0.1]) by localhost (smtp1.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KfYtFDcRwcjK for ; Thu, 17 Jun 2021 17:43:09 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by smtp1.osuosl.org (Postfix) with ESMTPS id 3A4D983DB6 for ; Thu, 17 Jun 2021 17:43:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1623951788; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/zzpM0uHejg0gMG9CjYapBiPLUutctpuSDUqVZ+2Q4E=; b=a791hgxxnFelQvYU0YGEoYGPTHNZf8lUroWx/pKH6hHC0bCwtz1PSlzTgbmy4imqVMvSBH 3lCmUB2pWFQBPm5rCnGwbzhcqGs2mOO1MgN4Xnp7tmLTIRlqBwIzNajx3dp+N0jI7v0C4V jpyjtEKjoLbe0FSH4Pj8ieWCbzpMWjQ= Received: from mail-pj1-f71.google.com (mail-pj1-f71.google.com [209.85.216.71]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-551-bBHmXGqNOf6pkUJn05VZ2w-1; Thu, 17 Jun 2021 13:43:07 -0400 X-MC-Unique: bBHmXGqNOf6pkUJn05VZ2w-1 Received: by mail-pj1-f71.google.com with SMTP id kz9-20020a17090b2109b029016f15e2b069so3571975pjb.6 for ; Thu, 17 Jun 2021 10:43:07 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/zzpM0uHejg0gMG9CjYapBiPLUutctpuSDUqVZ+2Q4E=; b=hQKrr+bBuPUkVCvBCXiZo5ivcZVEbL+BGNgSSjL8cBUHHKDlPeFFazxqT9DNIM6lqf NW57zGUaCNFpUX9W0tE+sZnyczbBGDR70nHRm8g5XWeOacbTYE1Ex/duFSrn47VLX7TP VuZw9mt0CzL0ci+T1wFhej2rm6h1yd/jKi+kWq+kKVJwupaOgXoJO23KbdAS/2pvv9HW uKSc5epqJJvbeQcWry//ApXAzItVzZGZf89n0u1XPBIAWQv3pSwdIOwI6pSuNVzq7gGT dDigDG5n0ub+cJCJCfajQeF+etE0GtBX8Hk8ekXmbhqqkCJ5GidpZRf+1IBINBQAzney jQXg== X-Gm-Message-State: AOAM5335V1XbDAeENdsrV1DGCp7+Wf04pHlvIU4GJ2BwQfG0/1lDYCAD 5Xwn5YjmbmjNx+fxxq00mmGFtTTqkHGG2eMUyaZVrPrPLmH8iKO7gQpp+2bgZk6kBkihSd4iQck P58H6a2Z2Cu0hmilk50WURt4eXuQ17Z+Z9P8LJmXFMxzJ+JolltWg9/MA+/ISuAr76w4v X-Received: by 2002:aa7:8392:0:b029:2db:4a7:d866 with SMTP id u18-20020aa783920000b02902db04a7d866mr918742pfm.66.1623951785953; Thu, 17 Jun 2021 10:43:05 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzwDU7iNtsM2HS/nq5yFznvsEs/ckmLn4U6yLNIxgWWsGZTF9BFZFkjsBQTitynf66DOKxDuw== X-Received: by 2002:aa7:8392:0:b029:2db:4a7:d866 with SMTP id u18-20020aa783920000b02902db04a7d866mr918726pfm.66.1623951785673; Thu, 17 Jun 2021 10:43:05 -0700 (PDT) Received: from wsfd-netdev91.ntdv.lab.eng.bos.redhat.com (nat-pool-bos-t.redhat.com. [66.187.233.206]) by smtp.gmail.com with ESMTPSA id y34sm5687873pfa.181.2021.06.17.10.43.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Jun 2021 10:43:05 -0700 (PDT) From: Mark Gray To: dev@openvswitch.org Date: Thu, 17 Jun 2021 13:42:52 -0400 Message-Id: <20210617174253.3459548-3-mark.d.gray@redhat.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210617174253.3459548-1-mark.d.gray@redhat.com> References: <20210617174253.3459548-1-mark.d.gray@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=mark.d.gray@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH ovn v2 2/3] ovn-northd: Add useful stopwatches X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" For performance measurement, it is useful to understand the length of time required to complete a number of key code paths in ovn-northd.c. Add stopwatches to measure these timings. Signed-off-by: Mark Gray --- northd/ovn-northd-ddlog.c | 15 +++++++++++++++ northd/ovn-northd.c | 20 ++++++++++++++++++++ 2 files changed, 35 insertions(+) diff --git a/northd/ovn-northd-ddlog.c b/northd/ovn-northd-ddlog.c index a4f2960bdcb8..7c552d516550 100644 --- a/northd/ovn-northd-ddlog.c +++ b/northd/ovn-northd-ddlog.c @@ -37,6 +37,7 @@ #include "ovsdb-parser.h" #include "ovsdb-types.h" #include "simap.h" +#include "stopwatch.h" #include "stream-ssl.h" #include "stream.h" #include "unixctl.h" @@ -50,6 +51,10 @@ VLOG_DEFINE_THIS_MODULE(ovn_northd); #include "northd/ovn-northd-ddlog-nb.inc" #include "northd/ovn-northd-ddlog-sb.inc" +#define NORTHD_LOOP_STOPWATCH_NAME "ovn-northd-loop" +#define OVNNB_DB_RUN_STOPWATCH_NAME "ovnnb_db_run" +#define OVNSB_DB_RUN_STOPWATCH_NAME "ovnsb_db_run" + struct northd_status { bool locked; bool pause; @@ -1259,6 +1264,10 @@ main(int argc, char *argv[]) daemonize_complete(); + stopwatch_create(NORTHD_LOOP_STOPWATCH_NAME, SW_MS); + stopwatch_create(OVNNB_DB_RUN_STOPWATCH_NAME, SW_MS); + stopwatch_create(OVNSB_DB_RUN_STOPWATCH_NAME, SW_MS); + /* Main loop. */ exiting = false; while (!exiting) { @@ -1285,8 +1294,12 @@ main(int argc, char *argv[]) status.locked = has_lock; status.pause = sb_ctx->paused; + stopwatch_start(OVNNB_DB_RUN_STOPWATCH_NAME, time_msec()); northd_run(nb_ctx); + stopwatch_stop(OVNNB_DB_RUN_STOPWATCH_NAME, time_msec()); + stopwatch_start(OVNSB_DB_RUN_STOPWATCH_NAME, time_msec()); northd_run(sb_ctx); + stopwatch_stop(OVNSB_DB_RUN_STOPWATCH_NAME, time_msec()); northd_update_probe_interval(nb_ctx, sb_ctx); if (ovsdb_cs_has_lock(sb_ctx->cs) && sb_ctx->state == S_UPDATE && @@ -1297,6 +1310,8 @@ main(int argc, char *argv[]) northd_send_deltas(sb_ctx); } + stopwatch_stop(NORTHD_LOOP_STOPWATCH_NAME, time_msec()); + stopwatch_start(NORTHD_LOOP_STOPWATCH_NAME, time_msec()); unixctl_server_run(unixctl); northd_wait(nb_ctx); diff --git a/northd/ovn-northd.c b/northd/ovn-northd.c index d872f6a3cc1d..bffa18de5c2d 100644 --- a/northd/ovn-northd.c +++ b/northd/ovn-northd.c @@ -49,6 +49,7 @@ #include "smap.h" #include "sset.h" #include "svec.h" +#include "stopwatch.h" #include "stream.h" #include "stream-ssl.h" #include "timeval.h" @@ -59,6 +60,10 @@ VLOG_DEFINE_THIS_MODULE(ovn_northd); +#define NORTHD_LOOP_STOPWATCH_NAME "ovn-northd-loop" +#define OVNNB_DB_RUN_STOPWATCH_NAME "ovnnb_db_run" +#define OVNSB_DB_RUN_STOPWATCH_NAME "ovnsb_db_run" + static unixctl_cb_func ovn_northd_exit; static unixctl_cb_func ovn_northd_pause; static unixctl_cb_func ovn_northd_resume; @@ -13238,6 +13243,9 @@ ovnnb_db_run(struct northd_context *ctx, if (!ctx->ovnsb_txn || !ctx->ovnnb_txn) { return; } + + stopwatch_start(OVNNB_DB_RUN_STOPWATCH_NAME, time_msec()); + struct hmap port_groups; struct hmap mcast_groups; struct hmap igmp_groups; @@ -13379,6 +13387,8 @@ ovnnb_db_run(struct northd_context *ctx, * as well. */ cleanup_macam(); + + stopwatch_stop(OVNNB_DB_RUN_STOPWATCH_NAME, time_msec()); } /* Stores the list of chassis which references an ha_chassis_group. @@ -13970,6 +13980,8 @@ ovnsb_db_run(struct northd_context *ctx, return; } + stopwatch_start(OVNSB_DB_RUN_STOPWATCH_NAME, time_msec()); + struct shash ha_ref_chassis_map = SHASH_INITIALIZER(&ha_ref_chassis_map); handle_port_binding_changes(ctx, ports, &ha_ref_chassis_map); update_northbound_cfg(ctx, sb_loop, loop_start_time); @@ -13977,6 +13989,8 @@ ovnsb_db_run(struct northd_context *ctx, update_sb_ha_group_ref_chassis(&ha_ref_chassis_map); } shash_destroy(&ha_ref_chassis_map); + + stopwatch_stop(OVNSB_DB_RUN_STOPWATCH_NAME, time_msec()); } static void @@ -14425,6 +14439,10 @@ main(int argc, char *argv[]) char *ovn_internal_version = ovn_get_internal_version(); VLOG_INFO("OVN internal version is : [%s]", ovn_internal_version); + stopwatch_create(NORTHD_LOOP_STOPWATCH_NAME, SW_MS); + stopwatch_create(OVNNB_DB_RUN_STOPWATCH_NAME, SW_MS); + stopwatch_create(OVNSB_DB_RUN_STOPWATCH_NAME, SW_MS); + /* Main loop. */ exiting = false; @@ -14508,6 +14526,8 @@ main(int argc, char *argv[]) ovsdb_idl_wait(ovnsb_idl_loop.idl); } + stopwatch_stop(NORTHD_LOOP_STOPWATCH_NAME, time_msec()); + stopwatch_start(NORTHD_LOOP_STOPWATCH_NAME, time_msec()); unixctl_server_run(unixctl); unixctl_server_wait(unixctl); memory_wait(); From patchwork Thu Jun 17 17:42:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Mark Gray X-Patchwork-Id: 1493780 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=openvswitch.org (client-ip=2605:bc80:3010::133; helo=smtp2.osuosl.org; envelope-from=ovs-dev-bounces@openvswitch.org; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=CN88iSIL; dkim-atps=neutral Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4G5Ttk1v1sz9sCD for ; Fri, 18 Jun 2021 03:43:41 +1000 (AEST) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 0FABE40604; Thu, 17 Jun 2021 17:43:39 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id m6pjI6jou7bz; Thu, 17 Jun 2021 17:43:37 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [IPv6:2605:bc80:3010:104::8cd3:938]) by smtp2.osuosl.org (Postfix) with ESMTPS id C9A6B405A8; Thu, 17 Jun 2021 17:43:36 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 887F3C000E; Thu, 17 Jun 2021 17:43:36 +0000 (UTC) X-Original-To: dev@openvswitch.org Delivered-To: ovs-dev@lists.linuxfoundation.org Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id 17AD1C000B for ; Thu, 17 Jun 2021 17:43:36 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id 4EDBB405A8 for ; Thu, 17 Jun 2021 17:43:20 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 7ljAlCyot7SK for ; Thu, 17 Jun 2021 17:43:16 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.8.0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by smtp2.osuosl.org (Postfix) with ESMTPS id D75CA405F3 for ; Thu, 17 Jun 2021 17:43:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1623951794; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+10dK/2mNTtcXb/r84cRY3Gst+vWJFLbvidWAJH4qqU=; b=CN88iSILxe5QaS+A0eczDxuU0qGorUfAgXHRONp0FnJcQWsH9EbOYjAQqTDGzLR0Q/h3lr UaqzmESJR2XS/mL8gobDRJtbieefh2dZ06huulip2zn4BNGv8Mc1NMWDXbPZ/COp3VmILi jrsgyZbsXNc9CI0P6lN3lKdkVwRazT8= Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-515-OQx0UuYyPhaPwhuOmNcJOw-1; Thu, 17 Jun 2021 13:43:10 -0400 X-MC-Unique: OQx0UuYyPhaPwhuOmNcJOw-1 Received: by mail-pl1-f199.google.com with SMTP id s2-20020a170902ea02b0290120de818cb9so465958plg.18 for ; Thu, 17 Jun 2021 10:43:10 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Zx/CLD8v8TJngnl0UrDFVWGdgh1VLzdvo+Y7HU968xU=; b=hqSRiIrlMnj+fcngIYplL/MsaCOd8NUCjc+Yo5f9EgfZWO8FuTHX+am5Gl/r6kjdco ELFLH+A4ZdNuDsz7x1k5a12mfsoMWj33VQs6PMF3n7D5//j0Ph9kZGa1vm6Bq6qCdLeG GlaPa1rnAuPiAU9tU9RaCZxZPPNkCP4H3yBp5xdKJJRVtHKVPBl7pyIl9maRVIB3sZvK 7+I6lpBQhknvPwffibA0pu780yI9Z2s9SH1a4naIWQx6aVERRXrkUoyUm3zAmdryo+rA nRXmEKtYaunLHys+gI+0c5nlBN/ATFnVoEOZJweif0staz/megMkmVWJ04posAG3XKIU x7GQ== X-Gm-Message-State: AOAM532pAGBzY0NSZ63GW56KYfYa1vLnw2bXkcFe6BMxcJ6eXiHzYkNo GkSB1t+S7dEdNq74H+RBTjI9PYa1RhO6k9wAEHctN3q/YmXjjmhlNNkNrmKP3jE9f9TC4rdWMh6 M0GAcUE/mhCu7tezcD7ekDjMMKMYohowMALKNvltRAupuKKu587pe43zl40g6UQusegR4 X-Received: by 2002:a65:610c:: with SMTP id z12mr6084826pgu.453.1623951789333; Thu, 17 Jun 2021 10:43:09 -0700 (PDT) X-Google-Smtp-Source: ABdhPJxNOhPD1tfxiSpilkDEKBAyueYJ0x7OGpsie7+GGD5KU9GXKb3XXdBzCKAMu1eD8+heF27tkA== X-Received: by 2002:a65:610c:: with SMTP id z12mr6084788pgu.453.1623951788834; Thu, 17 Jun 2021 10:43:08 -0700 (PDT) Received: from wsfd-netdev91.ntdv.lab.eng.bos.redhat.com (nat-pool-bos-t.redhat.com. [66.187.233.206]) by smtp.gmail.com with ESMTPSA id y34sm5687873pfa.181.2021.06.17.10.43.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Jun 2021 10:43:08 -0700 (PDT) From: Mark Gray To: dev@openvswitch.org Date: Thu, 17 Jun 2021 13:42:53 -0400 Message-Id: <20210617174253.3459548-4-mark.d.gray@redhat.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210617174253.3459548-1-mark.d.gray@redhat.com> References: <20210617174253.3459548-1-mark.d.gray@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=mark.d.gray@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Subject: [ovs-dev] [PATCH ovn v2 3/3] tests: Add check-perf target X-BeenThere: ovs-dev@openvswitch.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: ovs-dev-bounces@openvswitch.org Sender: "dev" Add a suite of micro-benchmarks to aid a developer in understanding the performance impact of any changes that they are making. They can be used to help to understand the relative performance between two test runs on the same test machine, but are not intended to give the absolute performance of OVN. To invoke the performance testsuite, run: $ make check-perf This will run all available performance tests. Additional metrics (e.g. memory, coverage, perf counters) may be added in the future. Additional tests (e.g. additional topologies, ovn-controller tests) may be added in the future. Signed-off-by: Mark Gray --- Notes: v2: create results directory to fix build error Documentation/topics/testing.rst | 49 ++++++++ tests/.gitignore | 3 + tests/automake.mk | 26 ++++ tests/perf-northd.at | 207 +++++++++++++++++++++++++++++++ tests/perf-testsuite.at | 26 ++++ 5 files changed, 311 insertions(+) create mode 100644 tests/perf-northd.at create mode 100644 tests/perf-testsuite.at diff --git a/Documentation/topics/testing.rst b/Documentation/topics/testing.rst index be9e7c57331c..ccd3278437b1 100644 --- a/Documentation/topics/testing.rst +++ b/Documentation/topics/testing.rst @@ -256,3 +256,52 @@ the following:: All the features documented under `Unit Tests`_ are available for the datapath testsuites, except that the datapath testsuites do not support running tests in parallel. + +Performance testing +~~~~~~~~~~~~~~~~~~~ + +OVN includes a suite of micro-benchmarks to aid a developer in understanding the +performance impact of any changes that they are making. They can be used to +help to understand the relative performance between two test runs on the same +test machine, but are not intended to give the absolute performance of OVN. + +To invoke the performance testsuite, run:: + + $ make check-perf + +This will run all available performance tests. Some of these tests may be +long-running as they need to build complex logical network topologies. In order +to speed up subsequent test runs, some objects (e.g. the Northbound DB) may be +cached. In order to force the tests to rebuild all these objects, run:: + + $ make check-perf TESTSUITEFLAGS="--rebuild" + +A typical workflow for a developer trying to improve the performance of OVN +would be the following: + +0. Optional: Modify/add a performance test to buld the topology that you are + benchmarking, if required. +1. Run ``make check-perf TESTSUITEFLAGS="--rebuild"`` to generate cached + databases. + +.. note:: + This step may take some time depending on the number of tests that are being + rebuilt, the complexity of the tests and the performance of the test machine. + If you are only using one test, you can specify the test to run by adding the + test number to the ``make`` command. + (e.g. ``make check-perf TESTSUITEFLAGS="--rebuild "``) + +2. Run ``make check-perf`` to measure the performance metric that you are + benchmarking against. If you are only using one test, you can specify the test to run by adding the test number to the ``make`` command. + (e.g. ``make check-perf TESTSUITEFLAGS="--rebuild "``) +3. Modify OVN code to implement the change that you believe will improve the + performance. +4. Go to Step 2. to continue making improvements. + +If, as a developer, you modify a performance test in a way that may change one +of these cached objects, be sure to rebuild the test. + +The results of each test run are displayed on the screen at the end of the test +run but are also saved in the file ``tests/perf-testsuite.dir/results``. The +cached objects are stored under the relevant folder in +``tests/perf-testsuite.dir/cached``. diff --git a/tests/.gitignore b/tests/.gitignore index 8479f9bb0f8f..65cb1c6e4fad 100644 --- a/tests/.gitignore +++ b/tests/.gitignore @@ -22,6 +22,9 @@ /system-offloads-testsuite /system-offloads-testsuite.dir/ /system-offloads-testsuite.log +/perf-testsuite +/perf-testsuite.dir/ +/perf-testsuite.log /test-aes128 /test-atomic /test-bundle diff --git a/tests/automake.mk b/tests/automake.mk index 742e5cff28cc..ab04461eef69 100644 --- a/tests/automake.mk +++ b/tests/automake.mk @@ -4,9 +4,11 @@ EXTRA_DIST += \ $(SYSTEM_TESTSUITE_AT) \ $(SYSTEM_KMOD_TESTSUITE_AT) \ $(SYSTEM_USERSPACE_TESTSUITE_AT) \ + $(PERF_TESTSUITE_AT) \ $(TESTSUITE) \ $(SYSTEM_KMOD_TESTSUITE) \ $(SYSTEM_USERSPACE_TESTSUITE) \ + $(PERF_TESTSUITE) \ tests/atlocal.in \ $(srcdir)/package.m4 \ $(srcdir)/tests/testsuite \ @@ -52,6 +54,10 @@ SYSTEM_TESTSUITE_AT = \ tests/system-ovn.at \ tests/system-ovn-kmod.at +PERF_TESTSUITE_AT = \ + tests/perf-testsuite.at \ + tests/perf-northd.at + check_SCRIPTS += tests/atlocal TESTSUITE = $(srcdir)/tests/testsuite @@ -59,6 +65,9 @@ TESTSUITE_PATCH = $(srcdir)/tests/testsuite.patch TESTSUITE_DIR = $(abs_top_builddir)/tests/testsuite.dir SYSTEM_KMOD_TESTSUITE = $(srcdir)/tests/system-kmod-testsuite SYSTEM_USERSPACE_TESTSUITE = $(srcdir)/tests/system-userspace-testsuite +PERF_TESTSUITE = $(srcdir)/tests/perf-testsuite +PERF_TESTSUITE_DIR = $(abs_top_builddir)/tests/perf-testsuite.dir +PERF_TESTSUITE_RESULTS = $(PERF_TESTSUITE_DIR)/results DISTCLEANFILES += tests/atconfig tests/atlocal AUTOTEST_PATH = $(ovs_builddir)/utilities:$(ovs_builddir)/vswitchd:$(ovs_builddir)/ovsdb:$(ovs_builddir)/vtep:tests:$(PTHREAD_WIN32_DIR_DLL):$(SSL_DIR):controller-vtep:northd:utilities:controller:ic @@ -171,6 +180,19 @@ check-system-userspace: all clean-local: test ! -f '$(TESTSUITE)' || $(SHELL) '$(TESTSUITE)' -C tests --clean + +check-perf: all + @echo > $(PERF_TESTSUITE_RESULTS) + set $(SHELL) '$(PERF_TESTSUITE)' -C tests AUTOTEST_PATH='$(AUTOTEST_PATH)'; \ + $(SUDO) "$$@" $(TESTSUITEFLAGS) -j1 || (test X'$(RECHECK)' = Xyes && $(SUDO) "$$@" --recheck) + @echo + @echo '## -------------------- ##' + @echo '## Performance Results ##' + @echo '## -------------------- ##' + @cat $(PERF_TESTSUITE_RESULTS) + @echo + @echo "Results can be found in $(PERF_TESTSUITE_RESULTS)" + AUTOTEST = $(AUTOM4TE) --language=autotest @@ -193,6 +215,10 @@ $(SYSTEM_USERSPACE_TESTSUITE): package.m4 $(SYSTEM_TESTSUITE_AT) $(SYSTEM_USERSP $(AM_V_GEN)$(AUTOTEST) -I '$(srcdir)' -o $@.tmp $@.at $(AM_V_at)mv $@.tmp $@ +$(PERF_TESTSUITE): package.m4 $(PERF_TESTSUITE_AT) $(COMMON_MACROS_AT) + $(AM_V_GEN)$(AUTOTEST) -I '$(srcdir)' -o $@.tmp $@.at + $(AM_V_at)mv $@.tmp $@ + # The `:;' works around a Bash 3.2 bug when the output is not writeable. $(srcdir)/package.m4: $(top_srcdir)/configure.ac $(AM_V_GEN):;{ \ diff --git a/tests/perf-northd.at b/tests/perf-northd.at new file mode 100644 index 000000000000..eb0c391c4650 --- /dev/null +++ b/tests/perf-northd.at @@ -0,0 +1,207 @@ +AT_BANNER([ovn-northd performance tests]) + +# CACHE_NBDB() +# +# Save the current northbound database for future test runs. +# +m4_define([CACHE_NBDB],[ + mkdir -p ${at_suite_dir}/cached/${at_group} + cp -f ${ovs_base}/ovn-nb/ovn-nb.db ${at_suite_dir}/cached/${at_group}/ +]) + +# BUILD_NBDB([COMMANDS]) +# +# Configure the northbound database using COMMANDS. +# +# BUILD_NBDB() will check if there is a cached nortbound database present +# for this test group. If present, it will use the cached version. If the +# testsuite was run using the '--rebuild' flag, it will force a rebuild of the +# northbound database. +# +m4_define([BUILD_NBDB],[ + if [[ ! -f ${at_suite_dir}/cached/${at_group}/ovn-nb.db ]] || [[ $at_arg_rebuild != false ]]; then + echo "Rebuild NBDB" + $1 + CACHE_NBDB() + else + echo "Using cached NBDB" + rm ${at_suite_dir}/cached/${at_group}/.ovn-nb.db.~lock~ + ovs-appctl -t ovn-nb/ovsdb-server ovsdb-server/remove-db OVN_Northbound + ovs-appctl -t ovn-nb/ovsdb-server ovsdb-server/add-db ${at_suite_dir}/cached/${at_group}/ovn-nb.db + fi + ovn-nbctl --wait=sb sync +]) + +# PERF_RECORD_BANNER() +# +# Append standard banner to performance results. +# +m4_define([PERF_RECORD_START],[ + echo >> ${at_suite_dir}/results + echo "$at_desc_line" >> ${at_suite_dir}/results + echo " ---" >> ${at_suite_dir}/results +]) + +# PERF_RECORD_RESULT([KEY], [VALUE]) +# +# Append KEY and VALUE to performance results. +# +m4_define([PERF_RECORD_RESULT],[ + echo " $1: $2" >> ${at_suite_dir}/results +]) + +m4_define([PARSE_STOPWATCH], [ + grep -A 6 $1 | grep $2 | sed 's/[[^0-9.]]*//g' +]) + +# PERF_RECORD_STOPWATCH([NAME], [METRIC]) +# +# Append the value of the OVN stopwatch metric METRIC from stopwatch NAME +# to performance results. +# +m4_define([PERF_RECORD_STOPWATCH], [ + PERF_RECORD_RESULT($3, [`ovn-appctl -t northd/NORTHD_TYPE stopwatch/show | PARSE_STOPWATCH($1, $2)`]) +]) + +# PERF_RECORD() +# +# Append a number of metrics to performance results +# +m4_define([PERF_RECORD_STOP], [ + PERF_RECORD_STOPWATCH(ovnnb_db_run, ["Maximum"], [Maximum (NB)]) + PERF_RECORD_STOPWATCH(ovnnb_db_run, ["Short term average"], [Average (NB)]) +]) + +# OVN_NBCTL([NBCTL_COMMAND]) +# +# Add NBCTL_COMMAND to list of commands to be run by RUN_OVN_NBCTL(). +# +m4_define([OVN_NBCTL], [ + command="${command} -- $1" +]) + +# RUN_OVN_NBCTL() +# +# Execute list of commands built by the OVN_NBCTL() macro. +# +m4_define([RUN_OVN_NBCTL], [ + check ovn-nbctl ${command} + unset command +]) + +OVS_START_SHELL_HELPERS +generate_subnet () { + local a=$(printf %d $(expr $1 / 256 + 10)) + local b=$(printf %d $(expr $1 % 256)) + echo $a.$b.0.0/16 +} +generate_ip () { + local a=$(printf %d $(expr $1 / 256 + 10)) + local b=$(printf %d $(expr $1 % 256)) + local c=$(printf %d $(expr $2 / 256)) + local d=$(printf %d $(expr $2 % 256)) + echo $a.$b.$c.$d +} +generate_router_ip () { + local a=$(printf %d $(expr $1 / 256 + 10)) + local b=$(printf %d $(expr $1 % 256)) + echo $a.$b.255.254 +} +generate_mac () { + local a=$(printf %02x $(expr $1 / 256)) + local b=$(printf %02x $(expr $1 % 256)) + local c=$(printf %02x $(expr $2 / 256)) + local d=$(printf %02x $(expr $2 % 256)) + echo f0:00:$a:$b:$c:$d +} +OVS_END_SHELL_HELPERS + +# OVN_BASIC_SCALE_CONFIG(HYPERVISORS, PORTS) +# +# Configures OVN with HYPERVISORS x logical switches. Each logical +# switch has PORTS x logical ports and is connected to one +# Logical Router. The logical router hosts an snat entry for the subnet hosted +# on each logical switch. This is illustrated below. +# +# . +# . +# . +# . +# . +# . Logical Switch ───┐ +# │ │ +# Logical Port ───────────┤ │ +# │ │ +# Logical Port ───────────┼── Logical Switch ───┤ +# │ │ +# Logical Port ───────────┤ ├──── Logical Router +# │ │ (snat entries) +# . Logical Switch ───┤ +# . │ +# . . │ +# . │ +# . +# +m4_define([OVN_BASIC_SCALE_CONFIG], [ + # In order to speed up building the NBDB, we run `ovn-nbctl` in daemon mode. + # We also coalesce `ovn-nbctl` commands using the OVN_NBCTL() and + # RUN_OVN_NBCTL() macros + # + on_exit 'kill $(cat ovn-nbctl.pid)' + export OVN_NB_DAEMON=$(ovn-nbctl --pidfile --detach) + + for hv in $(seq 1 $1); do + echo "Adding hypervisor ${hv}" + total_hv=$1 + logical_switch=lsw${hv} + logical_router=lrw${hv} + logical_router_port=lr${hv}lrp0 + logical_switch_router_port=lr${hv}lsp0 + logical_router_ip=$(generate_router_ip ${hv}) + logical_router_nat_ip=$(generate_ip $((hv+total_hv)) 1) + logical_router_mac=$(generate_mac ${hv} 1) + logical_switch_subnet=$(generate_subnet ${hv}) + + OVN_NBCTL(ls-add $logical_switch) + OVN_NBCTL(set Logical_Switch $logical_switch other_config:subnet=${logical_switch_subnet}) + OVN_NBCTL(set Logical_Switch $logical_switch other_config:exclude_ips=$logical_router_ip) + + OVN_NBCTL(lr-add $logical_router) + OVN_NBCTL(lrp-add $logical_router $logical_router_port $logical_router_mac ${logical_router_ip}/16) + OVN_NBCTL(lr-nat-add $logical_router snat $logical_router_nat_ip $logical_switch_subnet) + OVN_NBCTL(lsp-add $logical_switch $logical_switch_router_port -- set Logical_Switch_Port $logical_switch_router_port type=router options:router-port=$logical_router_port addresses=dynamic) + + for port in $(seq 1 $2); do + logical_switch_port=lsw${hv}lsp${port} + OVN_NBCTL(lsp-add $logical_switch $logical_switch_port) + OVN_NBCTL(lsp-set-addresses $logical_switch_port dynamic) + done + + RUN_OVN_NBCTL() + + done +]) + +OVN_FOR_EACH_NORTHD([ +AT_SETUP([ovn-northd basic scale test -- 200 Hypervisors, 200 Logical Ports/Hypervisor]) +PERF_RECORD_START() + +ovn_start + +BUILD_NBDB(OVN_BASIC_SCALE_CONFIG(200, 200)) + +PERF_RECORD_STOP() +AT_CLEANUP +]) + +OVN_FOR_EACH_NORTHD([ +AT_SETUP([ovn-northd basic scale test -- 500 Hypervisors, 50 Logical Ports/Hypervisor]) +PERF_RECORD_START() + +ovn_start + +BUILD_NBDB(OVN_BASIC_SCALE_CONFIG(500, 50)) + +PERF_RECORD_STOP() +AT_CLEANUP +]) \ No newline at end of file diff --git a/tests/perf-testsuite.at b/tests/perf-testsuite.at new file mode 100644 index 000000000000..31cdc850c55f --- /dev/null +++ b/tests/perf-testsuite.at @@ -0,0 +1,26 @@ +AT_INIT + +AT_COPYRIGHT([Copyright (c) 2021 Red Hat, + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at: + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License.]) + +m4_ifdef([AT_COLOR_TESTS], [AT_COLOR_TESTS]) +AT_ARG_OPTION([rebuild], [Do not use cached versions of databases]) + +m4_include([tests/ovs-macros.at]) +m4_include([tests/ovsdb-macros.at]) +m4_include([tests/ofproto-macros.at]) +m4_include([tests/ovn-macros.at]) + +m4_include([tests/perf-northd.at]) +