From patchwork Fri Mar 5 15:51:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Palethorpe X-Patchwork-Id: 1448001 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.linux.it (client-ip=2001:1418:10:5::2; helo=picard.linux.it; envelope-from=ltp-bounces+incoming=patchwork.ozlabs.org@lists.linux.it; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=suse.com header.i=@suse.com header.a=rsa-sha256 header.s=susede1 header.b=uVX2wC7X; dkim-atps=neutral Received: from picard.linux.it (picard.linux.it [IPv6:2001:1418:10:5::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4DsXKj6PScz9sWF for ; Sat, 6 Mar 2021 02:51:53 +1100 (AEDT) Received: from picard.linux.it (localhost [IPv6:::1]) by picard.linux.it (Postfix) with ESMTP id E426B3C6DE0 for ; Fri, 5 Mar 2021 16:51:49 +0100 (CET) X-Original-To: ltp@lists.linux.it Delivered-To: ltp@picard.linux.it Received: from in-7.smtp.seeweb.it (in-7.smtp.seeweb.it [IPv6:2001:4b78:1:20::7]) by picard.linux.it (Postfix) with ESMTP id 69CF83C564E for ; Fri, 5 Mar 2021 16:51:42 +0100 (CET) Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by in-7.smtp.seeweb.it (Postfix) with ESMTPS id BDF9C20077D for ; Fri, 5 Mar 2021 16:51:41 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1614959501; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Kuu4UqfbCKvQPVl0UzuQp/k4e/3yt9yCXcQJHVthPzk=; b=uVX2wC7X+Dets8RxcLD3HEbjYwX2IOMDVD/3IBrjhHXlI65x1SNJ/BrgP/trubi67xFf7G v2MdZOVPp0uXiV9d+dViB7ey4yGfmhfImslhV57Tk2yPMaPr6eTWuCtNbE8FvfW6gs+0MG hoHCUfrNVldDoJ8vpEjhkt5JuZNg1Q0= Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id 29B96AD3B; Fri, 5 Mar 2021 15:51:41 +0000 (UTC) To: ltp@lists.linux.it Date: Fri, 5 Mar 2021 15:51:18 +0000 Message-Id: <20210305155123.18199-2-rpalethorpe@suse.com> X-Mailer: git-send-email 2.30.1 In-Reply-To: <20210305155123.18199-1-rpalethorpe@suse.com> References: <20210305155123.18199-1-rpalethorpe@suse.com> MIME-Version: 1.0 X-Virus-Scanned: clamav-milter 0.102.4 at in-7.smtp.seeweb.it X-Virus-Status: Clean X-Spam-Status: No, score=0.1 required=7.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on in-7.smtp.seeweb.it Subject: [LTP] [PATCH 1/6] fzsync: Add self test X-BeenThere: ltp@lists.linux.it X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux Test Project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Richard Palethorpe via ltp From: Richard Palethorpe Reply-To: Richard Palethorpe Cc: Richard Palethorpe Errors-To: ltp-bounces+incoming=patchwork.ozlabs.org@lists.linux.it Sender: "ltp" Add a validation test for the Fuzz Sync library. It has a range of data races which Fuzzy Sync must reproduce. This is much easier to setup and run than reproducing real kernel bugs. Signed-off-by: Richard Palethorpe Suggested-by: Cyril Hrubis --- lib/newlib_tests/.gitignore | 1 + lib/newlib_tests/test16.c | 2 + lib/newlib_tests/tst_fuzzy_sync01.c | 233 ++++++++++++++++++++++++++++ 3 files changed, 236 insertions(+) create mode 100644 lib/newlib_tests/tst_fuzzy_sync01.c diff --git a/lib/newlib_tests/.gitignore b/lib/newlib_tests/.gitignore index 6c2612259..bb235f433 100644 --- a/lib/newlib_tests/.gitignore +++ b/lib/newlib_tests/.gitignore @@ -40,3 +40,4 @@ tst_bool_expr test_macros01 test_macros02 test_macros03 +tst_fuzzy_sync01 \ No newline at end of file diff --git a/lib/newlib_tests/test16.c b/lib/newlib_tests/test16.c index 0d74e1eae..7a9b5f20e 100644 --- a/lib/newlib_tests/test16.c +++ b/lib/newlib_tests/test16.c @@ -9,6 +9,8 @@ * which they should take it in turns to update. */ +#define _GNU_SOURCE + #include #include "tst_test.h" #include "tst_safe_pthread.h" diff --git a/lib/newlib_tests/tst_fuzzy_sync01.c b/lib/newlib_tests/tst_fuzzy_sync01.c new file mode 100644 index 000000000..22b78977a --- /dev/null +++ b/lib/newlib_tests/tst_fuzzy_sync01.c @@ -0,0 +1,233 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * Copyright (c) 2021 Richard Palethorpe + */ +#define _GNU_SOURCE + +#include "tst_test.h" +#include "tst_fuzzy_sync.h" + +enum race_window_state { + /* We are in a race window which will cause behaviour that has + * very different timings to the target. We must avoid these + * race windows. */ + TOO_EARLY, + TOO_LATE, + /* We are not in the target race window, but the timings of + * this behaviour are similar to those of the target. */ + EARLY_OR_LATE, + /* We in the target race window */ + CRITICAL, +}; + +enum wait_with { + SPIN, + SLEEP, +}; + +struct window { + const enum wait_with type; + const int time; +}; + +struct race_shape { + const struct window too_early; + const struct window early; + const struct window critical; + const struct window late; + const struct window too_late; +}; + +struct scenario { + const char *const name; + const struct race_shape a; + const struct race_shape b; +}; + +static struct tst_fzsync_pair pair; + +static volatile enum race_window_state state; + +static const struct scenario races[] = { + { "No delay", + {{SPIN, 0 }, {SPIN, 0 }, {SPIN, 100 }, {SPIN, 0 }, {SPIN, 0 }}, + {{SPIN, 0 }, {SPIN, 0 }, {SPIN, 100 }, {SPIN, 0 }, {SPIN, 0 }}, + }, + { "Exit aligned", + {{SPIN, 0 }, {SLEEP, 1 }, {SPIN, 10 }, {SPIN, 10 }, {SPIN, 0 }}, + {{SPIN, 0 }, {SLEEP, 1 }, {SPIN, 1000 }, {SPIN, 10 }, {SPIN, 0 }}, + }, + { "Entry aligned", + {{SPIN, 0 }, {SPIN, 1 }, {SPIN, 10 }, {SLEEP, 10 }, {SPIN, 0 }}, + {{SPIN, 0 }, {SPIN, 1 }, {SPIN, 1000 }, {SLEEP, 1 }, {SPIN, 0 }}, + }, + { "delay A entry", + {{SPIN, 0 }, {SLEEP, 1 }, {SPIN, 10 }, {SPIN, 10 }, {SPIN, 0 }}, + {{SPIN, 0 }, {SPIN, 1 }, {SPIN, 1000 }, {SPIN, 1 }, {SPIN, 0 }}, + }, + { "delay B entry", + {{SPIN, 0 }, {SPIN, 1 }, {SPIN, 10 }, {SPIN, 10 }, {SPIN, 0 }}, + {{SPIN, 0 }, {SLEEP, 1 }, {SPIN, 1000 }, {SPIN, 1 }, {SPIN, 0 }}, + }, + { "delay A exit", + {{SPIN, 0 }, {SPIN, 1 }, {SPIN, 10 }, {SLEEP, 10 }, {SPIN, 0 }}, + {{SPIN, 0 }, {SPIN, 1 }, {SPIN, 1000 }, {SPIN, 1 }, {SPIN, 0 }}, + }, + { "delay B exit", + {{SPIN, 0 }, {SPIN, 1 }, {SPIN, 10 }, {SPIN, 10 }, {SPIN, 0 }}, + {{SPIN, 0 }, {SPIN, 1 }, {SPIN, 1000 }, {SLEEP, 1 }, {SPIN, 0 }}, + }, + { "too early", + {{SPIN, 10 }, {SLEEP, 1 }, {SPIN, 100 }, {SPIN, 100 }, {SPIN, 0 }}, + {{SLEEP, 1 }, {SPIN, 1000 }, {SPIN, 1000 }, {SPIN, 10 }, {SPIN, 0 }}, + }, + { "too late", + {{SPIN, 10000 }, {SLEEP, 1 }, {SPIN, 100 }, {SPIN, 100 }, {SPIN, 10 }}, + {{SPIN, 0 }, {SPIN, 1000 }, {SPIN, 1000 }, {SPIN, 1000 }, {SPIN, 10 }}, + }, +}; + +static void cleanup(void) +{ + tst_fzsync_pair_cleanup(&pair); +} + +static void setup(void) +{ + pair.min_samples = 10000; + + tst_fzsync_pair_init(&pair); +} + +static void delay(struct window w) +{ + struct timespec ts = { 0, w.time }; + volatile int time = w.time; + + if (pair.yield_in_wait) + time /= 100; + + switch(w.type) { + case SPIN: + while (time--) { + if (pair.yield_in_wait) + sched_yield(); + } + break; + case SLEEP: + nanosleep(&ts, NULL); + break; + } +} + +static void *worker(void *v) +{ + unsigned int i = *(unsigned int *)v; + const struct race_shape *r = &races[i].b; + + while (tst_fzsync_run_b(&pair)) { + state = EARLY_OR_LATE; + + tst_fzsync_start_race_b(&pair); + if (r->too_early.time) { + state = TOO_EARLY; + delay(r->too_early); + } + + state = EARLY_OR_LATE; + delay(r->early); + + state = CRITICAL; + delay(r->critical); + + state = EARLY_OR_LATE; + delay(r->late); + + if (r->too_late.time) { + state = TOO_LATE; + delay(r->too_late); + } + tst_fzsync_end_race_b(&pair); + } + + return NULL; +} + +static void run(unsigned int i) +{ + const struct race_shape *r = &races[i].a; + struct tst_fzsync_run_thread wrap_run_b = { + .func = worker, + .arg = &i, + }; + int too_early = 0, early_or_late = 0, critical = 0, too_late = 0; + enum race_window_state state_copy; + + + tst_fzsync_pair_reset(&pair, NULL); + SAFE_PTHREAD_CREATE(&pair.thread_b, 0, tst_fzsync_thread_wrapper, + &wrap_run_b); + + while (tst_fzsync_run_a(&pair)) { + tst_fzsync_start_race_a(&pair); + + delay(r->too_early); + state_copy = state; + + switch(state_copy) { + case TOO_EARLY: + case TOO_LATE: + break; + default: + delay(r->early); + state_copy = state; + } + + switch(state_copy) { + case TOO_EARLY: + too_early++; + break; + case EARLY_OR_LATE: + early_or_late++; + delay(r->late); + break; + case CRITICAL: + critical++; + delay(r->critical); + break; + case TOO_LATE: + too_late++; + delay(r->too_late); + break; + default: + tst_brk(TBROK, "enum is not atomic?"); + } + tst_fzsync_end_race_a(&pair); + + switch(state_copy) { + case TOO_EARLY: + tst_fzsync_pair_add_bias(&pair, -10); + break; + case TOO_LATE: + tst_fzsync_pair_add_bias(&pair, 10); + break; + default: + ; + } + + if (critical > 100) { + tst_fzsync_pair_cleanup(&pair); + break; + } + } + + tst_res(critical ? TPASS : TFAIL, "%20s) =%-4d ~%-4d -%-4d +%-4d", + races[i].name, critical, early_or_late, too_early, too_late); +} + +static struct tst_test test = { + .tcnt = ARRAY_SIZE(races), + .test = run, + .setup = setup, + .cleanup = cleanup, +}; From patchwork Fri Mar 5 15:51:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Palethorpe X-Patchwork-Id: 1448002 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.linux.it (client-ip=213.254.12.146; helo=picard.linux.it; envelope-from=ltp-bounces+incoming=patchwork.ozlabs.org@lists.linux.it; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=suse.com header.i=@suse.com header.a=rsa-sha256 header.s=susede1 header.b=sIqIM4V3; dkim-atps=neutral Received: from picard.linux.it (picard.linux.it [213.254.12.146]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4DsXKr5qDPz9sWF for ; Sat, 6 Mar 2021 02:52:00 +1100 (AEDT) Received: from picard.linux.it (localhost [IPv6:::1]) by picard.linux.it (Postfix) with ESMTP id 75DA13C6DE7 for ; Fri, 5 Mar 2021 16:51:58 +0100 (CET) X-Original-To: ltp@lists.linux.it Delivered-To: ltp@picard.linux.it Received: from in-3.smtp.seeweb.it (in-3.smtp.seeweb.it [217.194.8.3]) by picard.linux.it (Postfix) with ESMTP id 242733C6DEF for ; Fri, 5 Mar 2021 16:51:46 +0100 (CET) Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by in-3.smtp.seeweb.it (Postfix) with ESMTPS id D6C891A002F0 for ; Fri, 5 Mar 2021 16:51:45 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1614959505; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vBp2OY65Nm8EVwQMwH5cS1ct0nRVIFZGG1Mov/zWHPw=; b=sIqIM4V3sd6SiwrYSLA2JMr4duk17pX9UHvp0BERC+RQJJAgOoDGlrVtJV0HNE4Zx02AJg L+soRe1ISszMgOWKxn6iJxqeUP7+bMOjHZa4rvVlacLa5WFKXaqlyf5WkyQj6EgAy/Gh1a BFv3zPSICyOMt8fE4qaOMdlNWWU5FfI= Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id 480F0ACBF; Fri, 5 Mar 2021 15:51:45 +0000 (UTC) To: ltp@lists.linux.it Date: Fri, 5 Mar 2021 15:51:19 +0000 Message-Id: <20210305155123.18199-3-rpalethorpe@suse.com> X-Mailer: git-send-email 2.30.1 In-Reply-To: <20210305155123.18199-1-rpalethorpe@suse.com> References: <20210305155123.18199-1-rpalethorpe@suse.com> MIME-Version: 1.0 X-Virus-Scanned: clamav-milter 0.102.4 at in-3.smtp.seeweb.it X-Virus-Status: Clean X-Spam-Status: No, score=0.1 required=7.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on in-3.smtp.seeweb.it Subject: [LTP] [PATCH 2/6] fzsync: Reset delay bias X-BeenThere: ltp@lists.linux.it X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux Test Project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Richard Palethorpe via ltp From: Richard Palethorpe Reply-To: Richard Palethorpe Cc: Richard Palethorpe Errors-To: ltp-bounces+incoming=patchwork.ozlabs.org@lists.linux.it Sender: "ltp" If the delay bias a preserved then it breaks tests which have multiple scenarios and/or use -i. The test author could reset it manually in this case, but it's likely to be error prone. Signed-off-by: Richard Palethorpe Reviewed-by: Cyril Hrubis --- include/tst_fuzzy_sync.h | 1 + 1 file changed, 1 insertion(+) diff --git a/include/tst_fuzzy_sync.h b/include/tst_fuzzy_sync.h index 4141f5c64..b8841d96d 100644 --- a/include/tst_fuzzy_sync.h +++ b/include/tst_fuzzy_sync.h @@ -289,6 +289,7 @@ static void tst_fzsync_pair_reset(struct tst_fzsync_pair *pair, tst_init_stat(&pair->diff_ab); tst_init_stat(&pair->spins_avg); pair->delay = 0; + pair->delay_bias = 0; pair->sampling = pair->min_samples; pair->exec_loop = 0; From patchwork Fri Mar 5 15:51:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Palethorpe X-Patchwork-Id: 1448003 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.linux.it (client-ip=213.254.12.146; helo=picard.linux.it; envelope-from=ltp-bounces+incoming=patchwork.ozlabs.org@lists.linux.it; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=suse.com header.i=@suse.com header.a=rsa-sha256 header.s=susede1 header.b=kaTQUBpo; dkim-atps=neutral Received: from picard.linux.it (picard.linux.it [213.254.12.146]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4DsXL05J5Vz9sWF for ; Sat, 6 Mar 2021 02:52:08 +1100 (AEDT) Received: from picard.linux.it (localhost [IPv6:::1]) by picard.linux.it (Postfix) with ESMTP id 4200A3C6561 for ; Fri, 5 Mar 2021 16:52:06 +0100 (CET) X-Original-To: ltp@lists.linux.it Delivered-To: ltp@picard.linux.it Received: from in-2.smtp.seeweb.it (in-2.smtp.seeweb.it [217.194.8.2]) by picard.linux.it (Postfix) with ESMTP id 4A4123C6DD9 for ; Fri, 5 Mar 2021 16:51:46 +0100 (CET) Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by in-2.smtp.seeweb.it (Postfix) with ESMTPS id 0EFE4601C48 for ; Fri, 5 Mar 2021 16:51:46 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1614959505; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XrTeGK8ZLzqi5+sLsj6JgeWOwCTK7rplfbNCCgZzmEc=; b=kaTQUBpo9GDRLK6zIACnHgmnnbeDdDJncIfRGKBBElBO1Lr2zs0a8fvi39bncxcmCEVle4 PmRbftTfGn13/7iUbQ+tyK+67yjVHAknRZjYyR9pUzx55fAobWfuuEdvJfjOoEfIv9kZ2n GzkEwoUCClQCcd71zWawEcVi1Lusxds= Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id 9D5EBAD3B; Fri, 5 Mar 2021 15:51:45 +0000 (UTC) To: ltp@lists.linux.it Date: Fri, 5 Mar 2021 15:51:20 +0000 Message-Id: <20210305155123.18199-4-rpalethorpe@suse.com> X-Mailer: git-send-email 2.30.1 In-Reply-To: <20210305155123.18199-1-rpalethorpe@suse.com> References: <20210305155123.18199-1-rpalethorpe@suse.com> MIME-Version: 1.0 X-Virus-Scanned: clamav-milter 0.102.4 at in-2.smtp.seeweb.it X-Virus-Status: Clean X-Spam-Status: No, score=0.1 required=7.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on in-2.smtp.seeweb.it Subject: [LTP] [PATCH 3/6] fzsync: Correctly print positive lower delay range bound X-BeenThere: ltp@lists.linux.it X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux Test Project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Richard Palethorpe via ltp From: Richard Palethorpe Reply-To: Richard Palethorpe Cc: Richard Palethorpe Errors-To: ltp-bounces+incoming=patchwork.ozlabs.org@lists.linux.it Sender: "ltp" If the magnitude of delay_bias is large enough then it can turn the lower bound positive. Signed-off-by: Richard Palethorpe Reviewed-by: Cyril Hrubis --- include/tst_fuzzy_sync.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/include/tst_fuzzy_sync.h b/include/tst_fuzzy_sync.h index b8841d96d..4063e95cb 100644 --- a/include/tst_fuzzy_sync.h +++ b/include/tst_fuzzy_sync.h @@ -518,9 +518,9 @@ static void tst_fzsync_pair_update(struct tst_fzsync_pair *pair) tst_res(TINFO, "Reached deviation ratios < %.2f, introducing randomness", pair->max_dev_ratio); - tst_res(TINFO, "Delay range is [-%d, %d]", - (int)(pair->diff_sb.avg / per_spin_time) + pair->delay_bias, - (int)(pair->diff_sa.avg / per_spin_time) - pair->delay_bias); + tst_res(TINFO, "Delay range is [%d, %d]", + -(int)(pair->diff_sb.avg / per_spin_time) + pair->delay_bias, + (int)(pair->diff_sa.avg / per_spin_time) + pair->delay_bias); tst_fzsync_pair_info(pair); pair->sampling = -1; } From patchwork Fri Mar 5 15:51:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Palethorpe X-Patchwork-Id: 1448004 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.linux.it (client-ip=2001:1418:10:5::2; helo=picard.linux.it; envelope-from=ltp-bounces+incoming=patchwork.ozlabs.org@lists.linux.it; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=suse.com header.i=@suse.com header.a=rsa-sha256 header.s=susede1 header.b=ZWfm6qv7; dkim-atps=neutral Received: from picard.linux.it (picard.linux.it [IPv6:2001:1418:10:5::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4DsXLC1f6Xz9sWF for ; Sat, 6 Mar 2021 02:52:19 +1100 (AEDT) Received: from picard.linux.it (localhost [IPv6:::1]) by picard.linux.it (Postfix) with ESMTP id 686AF3C6560 for ; Fri, 5 Mar 2021 16:52:16 +0100 (CET) X-Original-To: ltp@lists.linux.it Delivered-To: ltp@picard.linux.it Received: from in-6.smtp.seeweb.it (in-6.smtp.seeweb.it [217.194.8.6]) by picard.linux.it (Postfix) with ESMTP id CA3D73C6DDA for ; Fri, 5 Mar 2021 16:51:46 +0100 (CET) Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by in-6.smtp.seeweb.it (Postfix) with ESMTPS id 52CE3140126B for ; Fri, 5 Mar 2021 16:51:46 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1614959506; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+L04J8JM7GjxdHEae98d3KRSTcGbIar/kwpSslmp4+k=; b=ZWfm6qv7EN2h4WcYWFANPdYXUMho7leylC5ICBv9RzhvwPK19Wv5pUYjTZ0gNXwKOobg25 ByBtRNQ49n4tUe4Q8ahEuZjh8enHRfHk4yZS+3rFNCZNxkSttQXkBhbqTsdhMSi7VLRhYM DtK29SLTX/2bb/RQMoNfbOqKBQYyXA4= Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id E1A19AD72; Fri, 5 Mar 2021 15:51:45 +0000 (UTC) To: ltp@lists.linux.it Date: Fri, 5 Mar 2021 15:51:21 +0000 Message-Id: <20210305155123.18199-5-rpalethorpe@suse.com> X-Mailer: git-send-email 2.30.1 In-Reply-To: <20210305155123.18199-1-rpalethorpe@suse.com> References: <20210305155123.18199-1-rpalethorpe@suse.com> MIME-Version: 1.0 X-Virus-Scanned: clamav-milter 0.102.4 at in-6.smtp.seeweb.it X-Virus-Status: Clean X-Spam-Status: No, score=0.1 required=7.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on in-6.smtp.seeweb.it Subject: [LTP] [PATCH 4/6] fzsync: Add sched_yield for single core machine X-BeenThere: ltp@lists.linux.it X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux Test Project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Richard Palethorpe via ltp From: Richard Palethorpe Reply-To: Richard Palethorpe Errors-To: ltp-bounces+incoming=patchwork.ozlabs.org@lists.linux.it Sender: "ltp" From: Leo Yu-Chi Liang Fuzzy sync library uses spin waiting mechanism to implement thread barrier behavior, which would cause this test to be time-consuming on single core machine. Fix this by adding sched_yield in the spin waiting loop, so that the thread yields cpu as soon as it enters the waiting loop. Signed-off-by: Leo Yu-Chi Liang --- include/tst_fuzzy_sync.h | 34 ++++++++++++++++++++++++++-------- 1 file changed, 26 insertions(+), 8 deletions(-) diff --git a/include/tst_fuzzy_sync.h b/include/tst_fuzzy_sync.h index 4063e95cb..5474f81e3 100644 --- a/include/tst_fuzzy_sync.h +++ b/include/tst_fuzzy_sync.h @@ -59,12 +59,15 @@ * @sa tst_fzsync_pair */ -#include -#include #include -#include #include +#include +#include +#include +#include +#include #include "tst_atomic.h" +#include "tst_cpu.h" #include "tst_timer.h" #include "tst_safe_pthread.h" @@ -180,6 +183,15 @@ struct tst_fzsync_pair { int exec_loop; /** Internal; The second thread or 0 */ pthread_t thread_b; + /** + * Internal; The flag indicates single core machines or not + * + * If running on single core machines, it would take considerable + * amount of time to run fuzzy sync library. + * Thus call sched_yield to give up cpu to decrease the test time. + */ + bool yield_in_wait; + }; #define CHK(param, low, hi, def) do { \ @@ -206,6 +218,7 @@ static void tst_fzsync_pair_init(struct tst_fzsync_pair *pair) CHK(max_dev_ratio, 0, 1, 0.1); CHK(exec_time_p, 0, 1, 0.5); CHK(exec_loops, 20, INT_MAX, 3000000); + CHK(yield_in_wait, 0, 1, (tst_ncpus() <= 1)); } #undef CHK @@ -551,7 +564,8 @@ static void tst_fzsync_pair_update(struct tst_fzsync_pair *pair) */ static inline void tst_fzsync_pair_wait(int *our_cntr, int *other_cntr, - int *spins) + int *spins, + bool yield_in_wait) { if (tst_atomic_inc(other_cntr) == INT_MAX) { /* @@ -565,6 +579,8 @@ static inline void tst_fzsync_pair_wait(int *our_cntr, && tst_atomic_load(our_cntr) < INT_MAX) { if (spins) (*spins)++; + if(yield_in_wait) + sched_yield(); } tst_atomic_store(0, other_cntr); @@ -582,6 +598,8 @@ static inline void tst_fzsync_pair_wait(int *our_cntr, while (tst_atomic_load(our_cntr) < tst_atomic_load(other_cntr)) { if (spins) (*spins)++; + if(yield_in_wait) + sched_yield(); } } } @@ -594,7 +612,7 @@ static inline void tst_fzsync_pair_wait(int *our_cntr, */ static inline void tst_fzsync_wait_a(struct tst_fzsync_pair *pair) { - tst_fzsync_pair_wait(&pair->a_cntr, &pair->b_cntr, NULL); + tst_fzsync_pair_wait(&pair->a_cntr, &pair->b_cntr, NULL, pair->yield_in_wait); } /** @@ -605,7 +623,7 @@ static inline void tst_fzsync_wait_a(struct tst_fzsync_pair *pair) */ static inline void tst_fzsync_wait_b(struct tst_fzsync_pair *pair) { - tst_fzsync_pair_wait(&pair->b_cntr, &pair->a_cntr, NULL); + tst_fzsync_pair_wait(&pair->b_cntr, &pair->a_cntr, NULL, pair->yield_in_wait); } /** @@ -710,7 +728,7 @@ static inline void tst_fzsync_start_race_a(struct tst_fzsync_pair *pair) static inline void tst_fzsync_end_race_a(struct tst_fzsync_pair *pair) { tst_fzsync_time(&pair->a_end); - tst_fzsync_pair_wait(&pair->a_cntr, &pair->b_cntr, &pair->spins); + tst_fzsync_pair_wait(&pair->a_cntr, &pair->b_cntr, &pair->spins, pair->yield_in_wait); } /** @@ -741,7 +759,7 @@ static inline void tst_fzsync_start_race_b(struct tst_fzsync_pair *pair) static inline void tst_fzsync_end_race_b(struct tst_fzsync_pair *pair) { tst_fzsync_time(&pair->b_end); - tst_fzsync_pair_wait(&pair->b_cntr, &pair->a_cntr, &pair->spins); + tst_fzsync_pair_wait(&pair->b_cntr, &pair->a_cntr, &pair->spins, pair->yield_in_wait); } /** From patchwork Fri Mar 5 15:51:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Palethorpe X-Patchwork-Id: 1448005 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.linux.it (client-ip=2001:1418:10:5::2; helo=picard.linux.it; envelope-from=ltp-bounces+incoming=patchwork.ozlabs.org@lists.linux.it; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=suse.com header.i=@suse.com header.a=rsa-sha256 header.s=susede1 header.b=V4HWnNrf; dkim-atps=neutral Received: from picard.linux.it (picard.linux.it [IPv6:2001:1418:10:5::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4DsXLM0YdHz9sWQ for ; Sat, 6 Mar 2021 02:52:27 +1100 (AEDT) Received: from picard.linux.it (localhost [IPv6:::1]) by picard.linux.it (Postfix) with ESMTP id 4B1E93C567D for ; Fri, 5 Mar 2021 16:52:24 +0100 (CET) X-Original-To: ltp@lists.linux.it Delivered-To: ltp@picard.linux.it Received: from in-2.smtp.seeweb.it (in-2.smtp.seeweb.it [217.194.8.2]) by picard.linux.it (Postfix) with ESMTP id 10A263C6DDA for ; Fri, 5 Mar 2021 16:51:46 +0100 (CET) Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by in-2.smtp.seeweb.it (Postfix) with ESMTPS id 96BBB60045C for ; Fri, 5 Mar 2021 16:51:46 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1614959506; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FKDRuuOvIBMNWAkyf5tscRymn2/bcp50kCtSYp3tdZU=; b=V4HWnNrff9qCylkJsa04vdxE7DQHuFGzfSmzuMFPYYT3wz7F/dEACfkHeTqQKar1FiHoS0 Mhk4MeJrE7hZGWjsP7yrhuqQy3VxNnLRhWYOcZ90hvSXx8KO9RIwuxtwbNZ1lDzwzZXXDa qZBbEi3S5Ai5cmlJyuJ6wYGRBE29558= Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id 3A6D7ADCD; Fri, 5 Mar 2021 15:51:46 +0000 (UTC) To: ltp@lists.linux.it Date: Fri, 5 Mar 2021 15:51:22 +0000 Message-Id: <20210305155123.18199-6-rpalethorpe@suse.com> X-Mailer: git-send-email 2.30.1 In-Reply-To: <20210305155123.18199-1-rpalethorpe@suse.com> References: <20210305155123.18199-1-rpalethorpe@suse.com> MIME-Version: 1.0 X-Virus-Scanned: clamav-milter 0.102.4 at in-2.smtp.seeweb.it X-Virus-Status: Clean X-Spam-Status: No, score=0.1 required=7.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on in-2.smtp.seeweb.it Subject: [LTP] [PATCH 5/6] fzsync: Move yield check out of loop and add yield to delay X-BeenThere: ltp@lists.linux.it X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux Test Project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Richard Palethorpe via ltp From: Richard Palethorpe Reply-To: Richard Palethorpe Cc: Richard Palethorpe Errors-To: ltp-bounces+incoming=patchwork.ozlabs.org@lists.linux.it Sender: "ltp" During my testing I found no difference between having the branch inside the loop and outside. However looking at the generated assembly, it definitely does perform the branch inside the loop. This could have an effect on some platform with worse branch prediction. So I have moved the branch outside of the loop. Also I have added sched_yield to the delay loop. If we only have one CPU then it is not delaying anything unless the other process can progress. Signed-off-by: Richard Palethorpe Reviewed-by: Leo Yu-Chi Liang Reviewed-by: Cyril Hrubis --- include/tst_fuzzy_sync.h | 72 ++++++++++++++++++++++++++++++---------- 1 file changed, 54 insertions(+), 18 deletions(-) diff --git a/include/tst_fuzzy_sync.h b/include/tst_fuzzy_sync.h index 5474f81e3..36a604e13 100644 --- a/include/tst_fuzzy_sync.h +++ b/include/tst_fuzzy_sync.h @@ -183,9 +183,9 @@ struct tst_fzsync_pair { int exec_loop; /** Internal; The second thread or 0 */ pthread_t thread_b; - /** - * Internal; The flag indicates single core machines or not - * + /** + * The flag indicates single core machines or not + * * If running on single core machines, it would take considerable * amount of time to run fuzzy sync library. * Thus call sched_yield to give up cpu to decrease the test time. @@ -575,31 +575,53 @@ static inline void tst_fzsync_pair_wait(int *our_cntr, * line above before doing that. If we are in rear position * then our counter may already have been set to zero. */ - while (tst_atomic_load(our_cntr) > 0 - && tst_atomic_load(our_cntr) < INT_MAX) { - if (spins) - (*spins)++; - if(yield_in_wait) + if (yield_in_wait) { + while (tst_atomic_load(our_cntr) > 0 + && tst_atomic_load(our_cntr) < INT_MAX) { + if (spins) + (*spins)++; + sched_yield(); + } + } else { + while (tst_atomic_load(our_cntr) > 0 + && tst_atomic_load(our_cntr) < INT_MAX) { + if (spins) + (*spins)++; + } } + tst_atomic_store(0, other_cntr); /* * Once both counters have been set to zero the invariant * is restored and we can continue. */ - while (tst_atomic_load(our_cntr) > 1) - ; + if (yield_in_wait) { + while (tst_atomic_load(our_cntr) > 1) + sched_yield(); + } else { + while (tst_atomic_load(our_cntr) > 1) + ; + } } else { /* * If our counter is less than the other thread's we are ahead * of it and need to wait. */ - while (tst_atomic_load(our_cntr) < tst_atomic_load(other_cntr)) { - if (spins) - (*spins)++; - if(yield_in_wait) + if (yield_in_wait) { + while (tst_atomic_load(our_cntr) < + tst_atomic_load(other_cntr)) { + if (spins) + (*spins)++; sched_yield(); + } + } else { + while (tst_atomic_load(our_cntr) < + tst_atomic_load(other_cntr)) { + if (spins) + (*spins)++; + } } } } @@ -713,8 +735,15 @@ static inline void tst_fzsync_start_race_a(struct tst_fzsync_pair *pair) tst_fzsync_wait_a(pair); delay = pair->delay; - while (delay < 0) - delay++; + if (pair->yield_in_wait) { + while (delay < 0) { + sched_yield(); + delay++; + } + } else { + while (delay < 0) + delay++; + } tst_fzsync_time(&pair->a_start); } @@ -744,8 +773,15 @@ static inline void tst_fzsync_start_race_b(struct tst_fzsync_pair *pair) tst_fzsync_wait_b(pair); delay = pair->delay; - while (delay > 0) - delay--; + if (pair->yield_in_wait) { + while (delay > 0) { + sched_yield(); + delay--; + } + } else { + while (delay > 0) + delay--; + } tst_fzsync_time(&pair->b_start); } From patchwork Fri Mar 5 15:51:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Richard Palethorpe X-Patchwork-Id: 1448006 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=lists.linux.it (client-ip=2001:1418:10:5::2; helo=picard.linux.it; envelope-from=ltp-bounces+incoming=patchwork.ozlabs.org@lists.linux.it; receiver=) Authentication-Results: ozlabs.org; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=suse.com header.i=@suse.com header.a=rsa-sha256 header.s=susede1 header.b=proGICww; dkim-atps=neutral Received: from picard.linux.it (picard.linux.it [IPv6:2001:1418:10:5::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 4DsXLV5VW5z9sWF for ; Sat, 6 Mar 2021 02:52:34 +1100 (AEDT) Received: from picard.linux.it (localhost [IPv6:::1]) by picard.linux.it (Postfix) with ESMTP id 2596B3C6562 for ; Fri, 5 Mar 2021 16:52:32 +0100 (CET) X-Original-To: ltp@lists.linux.it Delivered-To: ltp@picard.linux.it Received: from in-2.smtp.seeweb.it (in-2.smtp.seeweb.it [217.194.8.2]) by picard.linux.it (Postfix) with ESMTP id 3A6C23C5651 for ; Fri, 5 Mar 2021 16:51:47 +0100 (CET) Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by in-2.smtp.seeweb.it (Postfix) with ESMTPS id F28AC601C44 for ; Fri, 5 Mar 2021 16:51:46 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1614959506; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1l3BIoNk5Vn+vIoItIjSsd1vQHo28jJSpg1urQoTAiQ=; b=proGICwwPoIQvNOzaRpDRy2KkAgC0s+r8wsr4MxalUziVczWZHhdb7j8BLqGJJ1BQeX8Z8 ps3S68U+jloRs8rCTSY8j/o/y/XgqLqulbcjBn+1Cp+mUTgz6/BRWh7YTJ5liAP3LyMoy6 UDN9vVVxBzC8qC0wA844abtOWvMYPEs= Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id 85A88AEC2; Fri, 5 Mar 2021 15:51:46 +0000 (UTC) To: ltp@lists.linux.it Date: Fri, 5 Mar 2021 15:51:23 +0000 Message-Id: <20210305155123.18199-7-rpalethorpe@suse.com> X-Mailer: git-send-email 2.30.1 In-Reply-To: <20210305155123.18199-1-rpalethorpe@suse.com> References: <20210305155123.18199-1-rpalethorpe@suse.com> MIME-Version: 1.0 X-Virus-Scanned: clamav-milter 0.102.4 at in-2.smtp.seeweb.it X-Virus-Status: Clean X-Spam-Status: No, score=0.1 required=7.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS autolearn=disabled version=3.4.4 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on in-2.smtp.seeweb.it Subject: [LTP] [PATCH 6/6] fzsync: Check processor affinity X-BeenThere: ltp@lists.linux.it X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux Test Project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-Patchwork-Original-From: Richard Palethorpe via ltp From: Richard Palethorpe Reply-To: Richard Palethorpe Cc: Richard Palethorpe Errors-To: ltp-bounces+incoming=patchwork.ozlabs.org@lists.linux.it Sender: "ltp" It is useful for testing Fuzzy Sync itself to set the CPU affinity to a single core. The current processes affinity does not effect tst_ncpus(), but we can get the affinity separately. Note that checking this still does not guarantee we will use yield when restricted to only one core. We would have to periodically probe which CPUs threads are running on until we detect more than one CPU. Signed-off-by: Richard Palethorpe Reviewed-by: Leo Yu-Chi Liang --- include/tst_fuzzy_sync.h | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/include/tst_fuzzy_sync.h b/include/tst_fuzzy_sync.h index 36a604e13..ea356ab44 100644 --- a/include/tst_fuzzy_sync.h +++ b/include/tst_fuzzy_sync.h @@ -59,9 +59,11 @@ * @sa tst_fzsync_pair */ +#define _GNU_SOURCE + #include #include -#include +#include "lapi/cpuset.h" #include #include #include @@ -213,12 +215,26 @@ struct tst_fzsync_pair { */ static void tst_fzsync_pair_init(struct tst_fzsync_pair *pair) { + long ncpus = tst_ncpus(); +#ifdef CPU_COUNT + size_t cpusz = CPU_ALLOC_SIZE(ncpus); + cpu_set_t *cpus = CPU_ALLOC(ncpus); + + if (sched_getaffinity(0, cpusz, cpus)) { + tst_res(TWARN | TERRNO, "sched_getaffinity(0, %zu, %zx)", + cpusz, (size_t)cpus); + } else { + ncpus = CPU_COUNT(cpus); + } + free(cpus); +#endif + CHK(avg_alpha, 0, 1, 0.25); CHK(min_samples, 20, INT_MAX, 1024); CHK(max_dev_ratio, 0, 1, 0.1); CHK(exec_time_p, 0, 1, 0.5); CHK(exec_loops, 20, INT_MAX, 3000000); - CHK(yield_in_wait, 0, 1, (tst_ncpus() <= 1)); + CHK(yield_in_wait, 0, 1, (ncpus <= 1)); } #undef CHK