| Message ID | 20200508232032.1974027-2-andriin@fb.com |
|---|---|
| State | Changes Requested |
| Delegated to: | BPF Maintainers |
| Headers | show |
| Series | Add benchmark runner and few benchmarks | expand |
On 5/8/20 4:20 PM, Andrii Nakryiko wrote: > While working on BPF ringbuf implementation, testing, and benchmarking, I've > developed a pretty generic and modular benchmark runner, which seems to be > generically useful, as I've already used it for one more purpose (testing > fastest way to trigger BPF program, to minimize overhead of in-kernel code). > > This patch adds generic part of benchmark runner and sets up Makefile for > extending it with more sets of benchmarks. > > Benchmarker itself operates by spinning up specified number of producer and > consumer threads, setting up interval timer sending SIGALARM signal to > application once a second. Every second, current snapshot with hits/drops > counters are collected and stored in an array. Drops are useful for > producer/consumer benchmarks in which producer might overwhelm consumers. > > Once test finishes after given amount of warm-up and testing seconds, mean and > stddev are calculated (ignoring warm-up results) and is printed out to stdout. > This setup seems to give consistent and accurate results. > > To validate behavior, I added two atomic counting tests: global and local. > For global one, all the producer threads are atomically incrementing same > counter as fast as possible. This, of course, leads to huge drop of > performance once there is more than one producer thread due to CPUs fighting > for the same memory location. > > Local counting, on the other hand, maintains one counter per each producer > thread, incremented independently. Once per second, all counters are read and > added together to form final "counting throughput" measurement. As expected, > such setup demonstrates linear scalability with number of producers (as long > as there are enough physical CPU cores, of course). See example output below. > Also, this setup can nicely demonstrate disastrous effects of false sharing, > if care is not taken to take those per-producer counters apart into > independent cache lines. > > Demo output shows global counter first with 1 producer, then with 4. Both > total and per-producer performance significantly drop. The last run is local > counter with 4 producers, demonstrating near-perfect scalability. > > $ ./bench -a -w1 -d2 -p1 count-global > Setting up benchmark 'count-global'... > Benchmark 'count-global' started. > Iter 0 ( 24.822us): hits 148.179M/s (148.179M/prod), drops 0.000M/s > Iter 1 ( 37.939us): hits 149.308M/s (149.308M/prod), drops 0.000M/s > Iter 2 (-10.774us): hits 150.717M/s (150.717M/prod), drops 0.000M/s > Iter 3 ( 3.807us): hits 151.435M/s (151.435M/prod), drops 0.000M/s > Summary: hits 150.488 ± 1.079M/s (150.488M/prod), drops 0.000 ± 0.000M/s > > $ ./bench -a -w1 -d2 -p4 count-global > Setting up benchmark 'count-global'... > Benchmark 'count-global' started. > Iter 0 ( 60.659us): hits 53.910M/s ( 13.477M/prod), drops 0.000M/s > Iter 1 (-17.658us): hits 53.722M/s ( 13.431M/prod), drops 0.000M/s > Iter 2 ( 5.865us): hits 53.495M/s ( 13.374M/prod), drops 0.000M/s > Iter 3 ( 0.104us): hits 53.606M/s ( 13.402M/prod), drops 0.000M/s > Summary: hits 53.608 ± 0.113M/s ( 13.402M/prod), drops 0.000 ± 0.000M/s > > $ ./bench -a -w1 -d2 -p4 count-local > Setting up benchmark 'count-local'... > Benchmark 'count-local' started. > Iter 0 ( 23.388us): hits 640.450M/s (160.113M/prod), drops 0.000M/s > Iter 1 ( 2.291us): hits 605.661M/s (151.415M/prod), drops 0.000M/s > Iter 2 ( -6.415us): hits 607.092M/s (151.773M/prod), drops 0.000M/s > Iter 3 ( -1.361us): hits 601.796M/s (150.449M/prod), drops 0.000M/s > Summary: hits 604.849 ± 2.739M/s (151.212M/prod), drops 0.000 ± 0.000M/s > > Signed-off-by: Andrii Nakryiko <andriin@fb.com> > --- > tools/testing/selftests/bpf/.gitignore | 1 + > tools/testing/selftests/bpf/Makefile | 13 +- > tools/testing/selftests/bpf/bench.c | 372 ++++++++++++++++++ > tools/testing/selftests/bpf/bench.h | 74 ++++ > .../selftests/bpf/benchs/bench_count.c | 91 +++++ > 5 files changed, 550 insertions(+), 1 deletion(-) > create mode 100644 tools/testing/selftests/bpf/bench.c > create mode 100644 tools/testing/selftests/bpf/bench.h > create mode 100644 tools/testing/selftests/bpf/benchs/bench_count.c > > diff --git a/tools/testing/selftests/bpf/.gitignore b/tools/testing/selftests/bpf/.gitignore > index 3ff031972975..1bb204cee853 100644 > --- a/tools/testing/selftests/bpf/.gitignore > +++ b/tools/testing/selftests/bpf/.gitignore > @@ -38,3 +38,4 @@ test_cpp > /bpf_gcc > /tools > /runqslower > +/bench > diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile > index 3d942be23d09..289fffbf975e 100644 > --- a/tools/testing/selftests/bpf/Makefile > +++ b/tools/testing/selftests/bpf/Makefile > @@ -77,7 +77,7 @@ TEST_PROGS_EXTENDED := with_addr.sh \ > # Compile but not part of 'make run_tests' > TEST_GEN_PROGS_EXTENDED = test_sock_addr test_skb_cgroup_id_user \ > flow_dissector_load test_flow_dissector test_tcp_check_syncookie_user \ > - test_lirc_mode2_user xdping test_cpp runqslower > + test_lirc_mode2_user xdping test_cpp runqslower bench > > TEST_CUSTOM_PROGS = urandom_read > > @@ -405,6 +405,17 @@ $(OUTPUT)/test_cpp: test_cpp.cpp $(OUTPUT)/test_core_extern.skel.h $(BPFOBJ) > $(call msg,CXX,,$@) > $(CXX) $(CFLAGS) $^ $(LDLIBS) -o $@ > > +# Benchmark runner > +$(OUTPUT)/bench_%.o: benchs/bench_%.c bench.h > + $(call msg,CC,,$@) > + $(CC) $(CFLAGS) -c $(filter %.c,$^) $(LDLIBS) -o $@ > +$(OUTPUT)/bench.o: bench.h > +$(OUTPUT)/bench: LDLIBS += -lm > +$(OUTPUT)/bench: $(OUTPUT)/bench.o \ > + $(OUTPUT)/bench_count.o > + $(call msg,BINARY,,$@) > + $(CC) $(LDFLAGS) -o $@ $(filter %.a %.o,$^) $(LDLIBS) > + > EXTRA_CLEAN := $(TEST_CUSTOM_PROGS) $(SCRATCH_DIR) \ > prog_tests/tests.h map_tests/tests.h verifier/tests.h \ > feature \ > diff --git a/tools/testing/selftests/bpf/bench.c b/tools/testing/selftests/bpf/bench.c > new file mode 100644 > index 000000000000..dddc97cd4db6 > --- /dev/null > +++ b/tools/testing/selftests/bpf/bench.c > @@ -0,0 +1,372 @@ > +// SPDX-License-Identifier: GPL-2.0 > +/* Copyright (c) 2020 Facebook */ > +#define _GNU_SOURCE > +#include <argp.h> > +#include <linux/compiler.h> > +#include <sys/time.h> > +#include <sched.h> > +#include <fcntl.h> > +#include <pthread.h> > +#include <sys/sysinfo.h> > +#include <sys/resource.h> > +#include <signal.h> > +#include "bench.h" > + > +struct env env = { > + .warmup_sec = 1, > + .duration_sec = 5, > + .affinity = false, > + .consumer_cnt = 1, > + .producer_cnt = 1, > +}; > + > +static int libbpf_print_fn(enum libbpf_print_level level, > + const char *format, va_list args) > +{ > + if (level == LIBBPF_DEBUG && !env.verbose) > + return 0; > + return vfprintf(stderr, format, args); > +} > + > +static int bump_memlock_rlimit(void) > +{ > + struct rlimit rlim_new = { > + .rlim_cur = RLIM_INFINITY, > + .rlim_max = RLIM_INFINITY, > + }; > + > + return setrlimit(RLIMIT_MEMLOCK, &rlim_new); > +} > + > +void setup_libbpf() > +{ > + int err; > + > + libbpf_set_print(libbpf_print_fn); > + > + err = bump_memlock_rlimit(); > + if (err) > + fprintf(stderr, "failed to increase RLIMIT_MEMLOCK: %d", err); > +} > + > +void hits_drops_report_progress(int iter, struct bench_res *res, long delta_ns) > +{ > + double hits_per_sec, drops_per_sec; > + double hits_per_prod; > + > + hits_per_sec = res->hits / 1000000.0 / (delta_ns / 1000000000.0); > + hits_per_prod = hits_per_sec / env.producer_cnt; > + drops_per_sec = res->drops / 1000000.0 / (delta_ns / 1000000000.0); > + > + printf("Iter %3d (%7.3lfus): ", > + iter, (delta_ns - 1000000000) / 1000.0); > + > + printf("hits %8.3lfM/s (%7.3lfM/prod), drops %8.3lfM/s\n", > + hits_per_sec, hits_per_prod, drops_per_sec); > +} > + > +void hits_drops_report_final(struct bench_res res[], int res_cnt) > +{ > + int i; > + double hits_mean = 0.0, drops_mean = 0.0; > + double hits_stddev = 0.0, drops_stddev = 0.0; > + > + for (i = 0; i < res_cnt; i++) { > + hits_mean += res[i].hits / 1000000.0 / (0.0 + res_cnt); > + drops_mean += res[i].drops / 1000000.0 / (0.0 + res_cnt); > + } > + > + if (res_cnt > 1) { > + for (i = 0; i < res_cnt; i++) { > + hits_stddev += (hits_mean - res[i].hits / 1000000.0) * > + (hits_mean - res[i].hits / 1000000.0) / > + (res_cnt - 1.0); > + drops_stddev += (drops_mean - res[i].drops / 1000000.0) * > + (drops_mean - res[i].drops / 1000000.0) / > + (res_cnt - 1.0); > + } > + hits_stddev = sqrt(hits_stddev); > + drops_stddev = sqrt(drops_stddev); > + } > + printf("Summary: hits %8.3lf \u00B1 %5.3lfM/s (%7.3lfM/prod), ", > + hits_mean, hits_stddev, hits_mean / env.producer_cnt); > + printf("drops %8.3lf \u00B1 %5.3lfM/s\n", > + drops_mean, drops_stddev); The unicode char \u00B1 (for +|-) may cause some old compiler (e.g., 4.8.5) warnings as it needs C99 mode. /data/users/yhs/work/net-next/tools/testing/selftests/bpf/bench.c:91:9: warning: universal character names are only valid in C++ and C99 [enabled by default] printf("Summary: hits %8.3lf \u00B1 %5.3lfM/s (%7.3lfM/prod), ", "+|-" is alternative, but not as good as \u00B1 indeed. Newer compiler may already have the default C99. Maybe we can just add C99 for build `bench`? > +} > + > +const char *argp_program_version = "benchmark"; > +const char *argp_program_bug_address = "<bpf@vger.kernel.org>"; > +const char argp_program_doc[] = > +"benchmark Generic benchmarking framework.\n" > +"\n" > +"This tool runs benchmarks.\n" > +"\n" > +"USAGE: benchmark <bench-name>\n" > +"\n" > +"EXAMPLES:\n" > +" # run 'count-local' benchmark with 1 producer and 1 consumer\n" > +" benchmark count-local\n" > +" # run 'count-local' with 16 producer and 8 consumer thread, pinned to CPUs\n" > +" benchmark -p16 -c8 -a count-local\n"; Some of the above global variables probably are statics. But I do not have a strong preference on this. > + > +static const struct argp_option opts[] = { > + { "list", 'l', NULL, 0, "List available benchmarks"}, > + { "duration", 'd', "SEC", 0, "Duration of benchmark, seconds"}, > + { "warmup", 'w', "SEC", 0, "Warm-up period, seconds"}, > + { "producers", 'p', "NUM", 0, "Number of producer threads"}, > + { "consumers", 'c', "NUM", 0, "Number of consumer threads"}, > + { "verbose", 'v', NULL, 0, "Verbose debug output"}, > + { "affinity", 'a', NULL, 0, "Set consumer/producer thread affinity"}, > + { "b2b", 'b', NULL, 0, "Back-to-back mode"}, > + { "rb-output", 10001, NULL, 0, "Set consumer/producer thread affinity"}, I did not see b2b and rb-output options are processed in this file. > + {}, > +}; > + > +static error_t parse_arg(int key, char *arg, struct argp_state *state) > +{ > + static int pos_args; > + > + switch (key) { > + case 'v': > + env.verbose = true; > + break; > + case 'l': > + env.list = true; > + break; > + case 'd': > + env.duration_sec = strtol(arg, NULL, 10); > + if (env.duration_sec <= 0) { > + fprintf(stderr, "Invalid duration: %s\n", arg); > + argp_usage(state); > + } > + break; > + case 'w': > + env.warmup_sec = strtol(arg, NULL, 10); > + if (env.warmup_sec <= 0) { > + fprintf(stderr, "Invalid warm-up duration: %s\n", arg); > + argp_usage(state); > + } > + break; > + case 'p': > + env.producer_cnt = strtol(arg, NULL, 10); > + if (env.producer_cnt <= 0) { > + fprintf(stderr, "Invalid producer count: %s\n", arg); > + argp_usage(state); > + } > + break; > + case 'c': > + env.consumer_cnt = strtol(arg, NULL, 10); > + if (env.consumer_cnt <= 0) { > + fprintf(stderr, "Invalid consumer count: %s\n", arg); > + argp_usage(state); > + } > + break; > + case 'a': > + env.affinity = true; > + break; > + case ARGP_KEY_ARG: > + if (pos_args++) { > + fprintf(stderr, > + "Unrecognized positional argument: %s\n", arg); > + argp_usage(state); > + } > + env.bench_name = strdup(arg); > + break; > + default: > + return ARGP_ERR_UNKNOWN; > + } > + return 0; > +} > + > +static void parse_cmdline_args(int argc, char **argv) > +{ > + static const struct argp argp = { > + .options = opts, > + .parser = parse_arg, > + .doc = argp_program_doc, > + }; > + if (argp_parse(&argp, argc, argv, 0, NULL, NULL)) > + exit(1); > + if (!env.list && !env.bench_name) { > + argp_help(&argp, stderr, ARGP_HELP_DOC, "bench"); > + exit(1); > + } > +} > + > +static void collect_measurements(long delta_ns); > + > +static __u64 last_time_ns; > +static void sigalarm_handler(int signo) > +{ > + long new_time_ns = get_time_ns(); > + long delta_ns = new_time_ns - last_time_ns; > + > + collect_measurements(delta_ns); > + > + last_time_ns = new_time_ns; > +} > + > +/* set up periodic 1-second timer */ > +static void setup_timer() > +{ > + static struct sigaction sigalarm_action = { > + .sa_handler = sigalarm_handler, > + }; > + struct itimerval timer_settings = {}; > + int err; > + > + last_time_ns = get_time_ns(); > + err = sigaction(SIGALRM, &sigalarm_action, NULL); > + if (err < 0) { > + fprintf(stderr, "failed to install SIGALARM handler: %d\n", -errno); > + exit(1); > + } > + timer_settings.it_interval.tv_sec = 1; > + timer_settings.it_value.tv_sec = 1; > + err = setitimer(ITIMER_REAL, &timer_settings, NULL); > + if (err < 0) { > + fprintf(stderr, "failed to arm interval timer: %d\n", -errno); > + exit(1); > + } > +} > + > +static void set_thread_affinity(pthread_t thread, int cpu) > +{ > + cpu_set_t cpuset; > + > + CPU_ZERO(&cpuset); > + CPU_SET(cpu, &cpuset); > + if (pthread_setaffinity_np(thread, sizeof(cpuset), &cpuset)) { > + fprintf(stderr, "setting affinity to CPU #%d failed: %d\n", > + cpu, errno); > + exit(1); > + } > +} > + > +static struct bench_state { > + int res_cnt; > + struct bench_res *results; > + pthread_t *consumers; > + pthread_t *producers; > +} state; > + > +const struct bench *bench = NULL; > + > +extern const struct bench bench_count_global; > +extern const struct bench bench_count_local; > + > +static const struct bench *benchs[] = { > + &bench_count_global, > + &bench_count_local, > +}; > + > +static void setup_benchmark() > +{ > + int i, err; > + > + if (!env.bench_name) { > + fprintf(stderr, "benchmark name is not specified\n"); > + exit(1); > + } > + > + for (i = 0; i < ARRAY_SIZE(benchs); i++) { > + if (strcmp(benchs[i]->name, env.bench_name) == 0) { > + bench = benchs[i]; > + break; > + } > + } > + if (!bench) { > + fprintf(stderr, "benchmark '%s' not found\n", env.bench_name); > + exit(1); > + } > + > + printf("Setting up benchmark '%s'...\n", bench->name); > + > + state.producers = calloc(env.producer_cnt, sizeof(*state.producers)); > + state.consumers = calloc(env.consumer_cnt, sizeof(*state.consumers)); > + state.results = calloc(env.duration_sec + env.warmup_sec + 2, > + sizeof(*state.results)); > + if (!state.producers || !state.consumers || !state.results) > + exit(1); > + > + if (bench->validate) > + bench->validate(); > + if (bench->setup) > + bench->setup(); > + > + for (i = 0; i < env.consumer_cnt; i++) { > + err = pthread_create(&state.consumers[i], NULL, > + bench->consumer_thread, (void *)(long)i); > + if (err) { > + fprintf(stderr, "failed to create consumer thread #%d: %d\n", > + i, -errno); > + exit(1); > + } > + if (env.affinity) > + set_thread_affinity(state.consumers[i], i); > + } > + for (i = 0; i < env.producer_cnt; i++) { > + err = pthread_create(&state.producers[i], NULL, > + bench->producer_thread, (void *)(long)i); > + if (err) { > + fprintf(stderr, "failed to create producer thread #%d: %d\n", > + i, -errno); > + exit(1); > + } > + if (env.affinity) > + set_thread_affinity(state.producers[i], > + env.consumer_cnt + i); Here, we bind consumer threads first and then producer threads, I think this is probably just arbitrary choice? In certain cases, I think people may want to have more advanced binding scenarios, e.g., for hyperthreading, binding consumer and producer on the same core or different cores etc. One possibility is to introduce -c option similar to taskset. If -c not supplied, you can have the current default. Otherwise, using -c list. > + } > + > + printf("Benchmark '%s' started.\n", bench->name); > +} > + > +static pthread_mutex_t bench_done_mtx = PTHREAD_MUTEX_INITIALIZER; > +static pthread_cond_t bench_done = PTHREAD_COND_INITIALIZER; > + > +static void collect_measurements(long delta_ns) { > + int iter = state.res_cnt++; > + struct bench_res *res = &state.results[iter]; > + > + bench->measure(res); > + > + if (bench->report_progress) > + bench->report_progress(iter, res, delta_ns); > + > + if (iter == env.duration_sec + env.warmup_sec) { > + pthread_mutex_lock(&bench_done_mtx); > + pthread_cond_signal(&bench_done); > + pthread_mutex_unlock(&bench_done_mtx); > + } > +} > + > +int main(int argc, char **argv) > +{ > + parse_cmdline_args(argc, argv); > + > + if (env.list) { > + int i; > + > + printf("Available benchmarks:\n"); > + for (i = 0; i < ARRAY_SIZE(benchs); i++) { > + printf("- %s\n", benchs[i]->name); > + } > + return 0; > + } > + > + setup_benchmark(); > + > + setup_timer(); > + > + pthread_mutex_lock(&bench_done_mtx); > + pthread_cond_wait(&bench_done, &bench_done_mtx); > + pthread_mutex_unlock(&bench_done_mtx); > + > + if (bench->report_final) > + /* skip first sample */ > + bench->report_final(state.results + env.warmup_sec, > + state.res_cnt - env.warmup_sec); > + > + return 0; > +} > + > diff --git a/tools/testing/selftests/bpf/bench.h b/tools/testing/selftests/bpf/bench.h > new file mode 100644 > index 000000000000..08aa0c5b1177 > --- /dev/null > +++ b/tools/testing/selftests/bpf/bench.h > @@ -0,0 +1,74 @@ > +/* SPDX-License-Identifier: GPL-2.0 */ > +#pragma once > +#include <stdlib.h> > +#include <stdbool.h> > +#include <linux/err.h> > +#include <errno.h> > +#include <unistd.h> > +#include <bpf/bpf.h> > +#include <bpf/libbpf.h> > +#include <math.h> > +#include <time.h> > +#include <sys/syscall.h> > + > +struct env { > + char *bench_name; > + int duration_sec; > + int warmup_sec; > + bool verbose; > + bool list; > + bool back2back; seems not used. > + bool affinity; > + int consumer_cnt; > + int producer_cnt; > +}; > + > +struct bench_res { > + long hits; > + long drops; > +}; > + > +struct bench { > + const char *name; > + void (*validate)(); > + void (*setup)(); > + void *(*producer_thread)(void *ctx); > + void *(*consumer_thread)(void *ctx); > + void (*measure)(struct bench_res* res); > + void (*report_progress)(int iter, struct bench_res* res, long delta_ns); > + void (*report_final)(struct bench_res res[], int res_cnt); > +}; > + > +struct counter { > + long value; > +} __attribute__((aligned(128))); > + > +extern struct env env; > +extern const struct bench *bench; > + > +void setup_libbpf(); > +void hits_drops_report_progress(int iter, struct bench_res *res, long delta_ns); > +void hits_drops_report_final(struct bench_res res[], int res_cnt); > + > +static inline __u64 get_time_ns() { > + struct timespec t; > + > + clock_gettime(CLOCK_MONOTONIC, &t); > + > + return (u64)t.tv_sec * 1000000000 + t.tv_nsec; > +} > + > +static inline void atomic_inc(long *value) > +{ > + (void)__atomic_add_fetch(value, 1, __ATOMIC_RELAXED); > +} > + > +static inline void atomic_add(long *value, long n) > +{ > + (void)__atomic_add_fetch(value, n, __ATOMIC_RELAXED); > +} > + > +static inline long atomic_swap(long *value, long n) > +{ > + return __atomic_exchange_n(value, n, __ATOMIC_RELAXED); > +} > diff --git a/tools/testing/selftests/bpf/benchs/bench_count.c b/tools/testing/selftests/bpf/benchs/bench_count.c > new file mode 100644 > index 000000000000..befba7a82643 > --- /dev/null > +++ b/tools/testing/selftests/bpf/benchs/bench_count.c > @@ -0,0 +1,91 @@ > +// SPDX-License-Identifier: GPL-2.0 > +/* Copyright (c) 2020 Facebook */ > +#include "bench.h" > + > +/* COUNT-GLOBAL benchmark */ > + > +static struct count_global_ctx { > + struct counter hits; > +} count_global_ctx; > + > +static void *count_global_producer(void *input) > +{ > + struct count_global_ctx *ctx = &count_global_ctx; > + > + while (true) { > + atomic_inc(&ctx->hits.value); > + } > + return NULL; > +} > + > +static void *count_global_consumer(void *input) > +{ > + return NULL; > +} > + > +static void count_global_measure(struct bench_res *res) > +{ > + struct count_global_ctx *ctx = &count_global_ctx; > + > + res->hits = atomic_swap(&ctx->hits.value, 0); > +} > + > +/* COUNT-local benchmark */ > + > +static struct count_local_ctx { > + struct counter *hits; > +} count_local_ctx; > + > +static void count_local_setup() > +{ > + struct count_local_ctx *ctx = &count_local_ctx; > + > + ctx->hits = calloc(env.consumer_cnt, sizeof(*ctx->hits)); > + if (!ctx->hits) > + exit(1); > +} > + > +static void *count_local_producer(void *input) > +{ > + struct count_local_ctx *ctx = &count_local_ctx; > + int idx = (long)input; > + > + while (true) { > + atomic_inc(&ctx->hits[idx].value); > + } > + return NULL; > +} > + > +static void *count_local_consumer(void *input) > +{ > + return NULL; > +} > + > +static void count_local_measure(struct bench_res *res) > +{ > + struct count_local_ctx *ctx = &count_local_ctx; > + int i; > + > + for (i = 0; i < env.producer_cnt; i++) { > + res->hits += atomic_swap(&ctx->hits[i].value, 0); > + } > +} > + > +const struct bench bench_count_global = { > + .name = "count-global", > + .producer_thread = count_global_producer, > + .consumer_thread = count_global_consumer, > + .measure = count_global_measure, > + .report_progress = hits_drops_report_progress, > + .report_final = hits_drops_report_final, > +}; > + > +const struct bench bench_count_local = { > + .name = "count-local", > + .setup = count_local_setup, > + .producer_thread = count_local_producer, > + .consumer_thread = count_local_consumer, > + .measure = count_local_measure, > + .report_progress = hits_drops_report_progress, > + .report_final = hits_drops_report_final, > +}; >
On Sat, May 9, 2020 at 10:10 AM Yonghong Song <yhs@fb.com> wrote: > > > > On 5/8/20 4:20 PM, Andrii Nakryiko wrote: > > While working on BPF ringbuf implementation, testing, and benchmarking, I've > > developed a pretty generic and modular benchmark runner, which seems to be > > generically useful, as I've already used it for one more purpose (testing > > fastest way to trigger BPF program, to minimize overhead of in-kernel code). > > > > This patch adds generic part of benchmark runner and sets up Makefile for > > extending it with more sets of benchmarks. > > > > Benchmarker itself operates by spinning up specified number of producer and > > consumer threads, setting up interval timer sending SIGALARM signal to > > application once a second. Every second, current snapshot with hits/drops > > counters are collected and stored in an array. Drops are useful for > > producer/consumer benchmarks in which producer might overwhelm consumers. > > > > Once test finishes after given amount of warm-up and testing seconds, mean and > > stddev are calculated (ignoring warm-up results) and is printed out to stdout. > > This setup seems to give consistent and accurate results. > > > > To validate behavior, I added two atomic counting tests: global and local. > > For global one, all the producer threads are atomically incrementing same > > counter as fast as possible. This, of course, leads to huge drop of > > performance once there is more than one producer thread due to CPUs fighting > > for the same memory location. > > > > Local counting, on the other hand, maintains one counter per each producer > > thread, incremented independently. Once per second, all counters are read and > > added together to form final "counting throughput" measurement. As expected, > > such setup demonstrates linear scalability with number of producers (as long > > as there are enough physical CPU cores, of course). See example output below. > > Also, this setup can nicely demonstrate disastrous effects of false sharing, > > if care is not taken to take those per-producer counters apart into > > independent cache lines. > > > > Demo output shows global counter first with 1 producer, then with 4. Both > > total and per-producer performance significantly drop. The last run is local > > counter with 4 producers, demonstrating near-perfect scalability. > > > > $ ./bench -a -w1 -d2 -p1 count-global > > Setting up benchmark 'count-global'... > > Benchmark 'count-global' started. > > Iter 0 ( 24.822us): hits 148.179M/s (148.179M/prod), drops 0.000M/s > > Iter 1 ( 37.939us): hits 149.308M/s (149.308M/prod), drops 0.000M/s > > Iter 2 (-10.774us): hits 150.717M/s (150.717M/prod), drops 0.000M/s > > Iter 3 ( 3.807us): hits 151.435M/s (151.435M/prod), drops 0.000M/s > > Summary: hits 150.488 ± 1.079M/s (150.488M/prod), drops 0.000 ± 0.000M/s > > > > $ ./bench -a -w1 -d2 -p4 count-global > > Setting up benchmark 'count-global'... > > Benchmark 'count-global' started. > > Iter 0 ( 60.659us): hits 53.910M/s ( 13.477M/prod), drops 0.000M/s > > Iter 1 (-17.658us): hits 53.722M/s ( 13.431M/prod), drops 0.000M/s > > Iter 2 ( 5.865us): hits 53.495M/s ( 13.374M/prod), drops 0.000M/s > > Iter 3 ( 0.104us): hits 53.606M/s ( 13.402M/prod), drops 0.000M/s > > Summary: hits 53.608 ± 0.113M/s ( 13.402M/prod), drops 0.000 ± 0.000M/s > > > > $ ./bench -a -w1 -d2 -p4 count-local > > Setting up benchmark 'count-local'... > > Benchmark 'count-local' started. > > Iter 0 ( 23.388us): hits 640.450M/s (160.113M/prod), drops 0.000M/s > > Iter 1 ( 2.291us): hits 605.661M/s (151.415M/prod), drops 0.000M/s > > Iter 2 ( -6.415us): hits 607.092M/s (151.773M/prod), drops 0.000M/s > > Iter 3 ( -1.361us): hits 601.796M/s (150.449M/prod), drops 0.000M/s > > Summary: hits 604.849 ± 2.739M/s (151.212M/prod), drops 0.000 ± 0.000M/s > > > > Signed-off-by: Andrii Nakryiko <andriin@fb.com> > > --- > > tools/testing/selftests/bpf/.gitignore | 1 + > > tools/testing/selftests/bpf/Makefile | 13 +- > > tools/testing/selftests/bpf/bench.c | 372 ++++++++++++++++++ > > tools/testing/selftests/bpf/bench.h | 74 ++++ > > .../selftests/bpf/benchs/bench_count.c | 91 +++++ > > 5 files changed, 550 insertions(+), 1 deletion(-) > > create mode 100644 tools/testing/selftests/bpf/bench.c > > create mode 100644 tools/testing/selftests/bpf/bench.h > > create mode 100644 tools/testing/selftests/bpf/benchs/bench_count.c > > [...] trimming is good :) > > + > > +void hits_drops_report_final(struct bench_res res[], int res_cnt) > > +{ > > + int i; > > + double hits_mean = 0.0, drops_mean = 0.0; > > + double hits_stddev = 0.0, drops_stddev = 0.0; > > + > > + for (i = 0; i < res_cnt; i++) { > > + hits_mean += res[i].hits / 1000000.0 / (0.0 + res_cnt); > > + drops_mean += res[i].drops / 1000000.0 / (0.0 + res_cnt); > > + } > > + > > + if (res_cnt > 1) { > > + for (i = 0; i < res_cnt; i++) { > > + hits_stddev += (hits_mean - res[i].hits / 1000000.0) * > > + (hits_mean - res[i].hits / 1000000.0) / > > + (res_cnt - 1.0); > > + drops_stddev += (drops_mean - res[i].drops / 1000000.0) * > > + (drops_mean - res[i].drops / 1000000.0) / > > + (res_cnt - 1.0); > > + } > > + hits_stddev = sqrt(hits_stddev); > > + drops_stddev = sqrt(drops_stddev); > > + } > > + printf("Summary: hits %8.3lf \u00B1 %5.3lfM/s (%7.3lfM/prod), ", > > + hits_mean, hits_stddev, hits_mean / env.producer_cnt); > > + printf("drops %8.3lf \u00B1 %5.3lfM/s\n", > > + drops_mean, drops_stddev); > > The unicode char \u00B1 (for +|-) may cause some old compiler (e.g., > 4.8.5) warnings as it needs C99 mode. > > /data/users/yhs/work/net-next/tools/testing/selftests/bpf/bench.c:91:9: > warning: universal character names are only valid in C++ and C99 > [enabled by default] > printf("Summary: hits %8.3lf \u00B1 %5.3lfM/s (%7.3lfM/prod), ", > > "+|-" is alternative, but not as good as \u00B1 indeed. Newer > compiler may already have the default C99. Maybe we can just add > C99 for build `bench`? I think I'm fine with ancient compiler emitting harmless warning for code under selftests/bpf, honestly... > > > +} > > + > > +const char *argp_program_version = "benchmark"; > > +const char *argp_program_bug_address = "<bpf@vger.kernel.org>"; > > +const char argp_program_doc[] = > > +"benchmark Generic benchmarking framework.\n" > > +"\n" > > +"This tool runs benchmarks.\n" > > +"\n" > > +"USAGE: benchmark <bench-name>\n" > > +"\n" > > +"EXAMPLES:\n" > > +" # run 'count-local' benchmark with 1 producer and 1 consumer\n" > > +" benchmark count-local\n" > > +" # run 'count-local' with 16 producer and 8 consumer thread, pinned to CPUs\n" > > +" benchmark -p16 -c8 -a count-local\n"; > > Some of the above global variables probably are statics. > But I do not have a strong preference on this. Oh, it's actually global variables argp library expects, they have to be global. > > > + > > +static const struct argp_option opts[] = { > > + { "list", 'l', NULL, 0, "List available benchmarks"}, > > + { "duration", 'd', "SEC", 0, "Duration of benchmark, seconds"}, > > + { "warmup", 'w', "SEC", 0, "Warm-up period, seconds"}, > > + { "producers", 'p', "NUM", 0, "Number of producer threads"}, > > + { "consumers", 'c', "NUM", 0, "Number of consumer threads"}, > > + { "verbose", 'v', NULL, 0, "Verbose debug output"}, > > + { "affinity", 'a', NULL, 0, "Set consumer/producer thread affinity"}, > > + { "b2b", 'b', NULL, 0, "Back-to-back mode"}, > > + { "rb-output", 10001, NULL, 0, "Set consumer/producer thread affinity"}, > > I did not see b2b and rb-output options are processed in this file. Slipped through the rebasing cracks from the future ringbuf benchmarks, will remove it. I also figured out a way to do this more modular anyways (child parsers in argp). > > > + {}, > > +}; > > + [...] > > + for (i = 0; i < env.consumer_cnt; i++) { > > + err = pthread_create(&state.consumers[i], NULL, > > + bench->consumer_thread, (void *)(long)i); > > + if (err) { > > + fprintf(stderr, "failed to create consumer thread #%d: %d\n", > > + i, -errno); > > + exit(1); > > + } > > + if (env.affinity) > > + set_thread_affinity(state.consumers[i], i); > > + } > > + for (i = 0; i < env.producer_cnt; i++) { > > + err = pthread_create(&state.producers[i], NULL, > > + bench->producer_thread, (void *)(long)i); > > + if (err) { > > + fprintf(stderr, "failed to create producer thread #%d: %d\n", > > + i, -errno); > > + exit(1); > > + } > > + if (env.affinity) > > + set_thread_affinity(state.producers[i], > > + env.consumer_cnt + i); > > Here, we bind consumer threads first and then producer threads, I think > this is probably just arbitrary choice? yep, almost arbitrary. Most of my cases have 1 consumer and >=1 producers, so it was convenient to have consumer pinned to same CPU, regardless of how many producers I have. > > In certain cases, I think people may want to have more advanced binding > scenarios, e.g., for hyperthreading, binding consumer and producer on > the same core or different cores etc. One possibility is to introduce > -c option similar to taskset. If -c not supplied, you can have > the current default. Otherwise, using -c list. > well, taskset's job is simpler, it takes a list of CPUs for single PID, if I understand correctly. Here we have many threads, each might have different CPU or even CPUs. But I agree that for some benchmarks it's going to be critical to control this precisely. Here's how I'm going to allows most flexibility without too much complexity. --prod-affinity 1,2,10-16,100 -- will specify a set of CPUs for producers. First producer will use CPU with least index form that list. Second will take second, and so on. If there are less CPUs provided than necessary - it's an error. If more - it's fine. Then for consumers will add independent --cons-affinity parameters, which will do the same for consumer threads. Having two independent lists will allow to test scenarios where we want producers and consumers to fight for the same CPU. Does this sound ok? > > + } > > + > > + printf("Benchmark '%s' started.\n", bench->name); > > +} > > + [...] > > diff --git a/tools/testing/selftests/bpf/bench.h b/tools/testing/selftests/bpf/bench.h > > new file mode 100644 > > index 000000000000..08aa0c5b1177 > > --- /dev/null > > +++ b/tools/testing/selftests/bpf/bench.h > > @@ -0,0 +1,74 @@ > > +/* SPDX-License-Identifier: GPL-2.0 */ > > +#pragma once > > +#include <stdlib.h> > > +#include <stdbool.h> > > +#include <linux/err.h> > > +#include <errno.h> > > +#include <unistd.h> > > +#include <bpf/bpf.h> > > +#include <bpf/libbpf.h> > > +#include <math.h> > > +#include <time.h> > > +#include <sys/syscall.h> > > + > > +struct env { > > + char *bench_name; > > + int duration_sec; > > + int warmup_sec; > > + bool verbose; > > + bool list; > > + bool back2back; > > seems not used. yep, cleaning up > > > + bool affinity; > > + int consumer_cnt; > > + int producer_cnt; > > +}; > > + [...]
On 5/11/20 8:29 PM, Andrii Nakryiko wrote: > On Sat, May 9, 2020 at 10:10 AM Yonghong Song <yhs@fb.com> wrote: >> >> >> >> On 5/8/20 4:20 PM, Andrii Nakryiko wrote: >>> While working on BPF ringbuf implementation, testing, and benchmarking, I've >>> developed a pretty generic and modular benchmark runner, which seems to be >>> generically useful, as I've already used it for one more purpose (testing >>> fastest way to trigger BPF program, to minimize overhead of in-kernel code). >>> >>> This patch adds generic part of benchmark runner and sets up Makefile for >>> extending it with more sets of benchmarks. >>> >>> Benchmarker itself operates by spinning up specified number of producer and >>> consumer threads, setting up interval timer sending SIGALARM signal to >>> application once a second. Every second, current snapshot with hits/drops >>> counters are collected and stored in an array. Drops are useful for >>> producer/consumer benchmarks in which producer might overwhelm consumers. >>> >>> Once test finishes after given amount of warm-up and testing seconds, mean and >>> stddev are calculated (ignoring warm-up results) and is printed out to stdout. >>> This setup seems to give consistent and accurate results. >>> >>> To validate behavior, I added two atomic counting tests: global and local. >>> For global one, all the producer threads are atomically incrementing same >>> counter as fast as possible. This, of course, leads to huge drop of >>> performance once there is more than one producer thread due to CPUs fighting >>> for the same memory location. >>> >>> Local counting, on the other hand, maintains one counter per each producer >>> thread, incremented independently. Once per second, all counters are read and >>> added together to form final "counting throughput" measurement. As expected, >>> such setup demonstrates linear scalability with number of producers (as long >>> as there are enough physical CPU cores, of course). See example output below. >>> Also, this setup can nicely demonstrate disastrous effects of false sharing, >>> if care is not taken to take those per-producer counters apart into >>> independent cache lines. >>> >>> Demo output shows global counter first with 1 producer, then with 4. Both >>> total and per-producer performance significantly drop. The last run is local >>> counter with 4 producers, demonstrating near-perfect scalability. >>> >>> $ ./bench -a -w1 -d2 -p1 count-global >>> Setting up benchmark 'count-global'... >>> Benchmark 'count-global' started. >>> Iter 0 ( 24.822us): hits 148.179M/s (148.179M/prod), drops 0.000M/s >>> Iter 1 ( 37.939us): hits 149.308M/s (149.308M/prod), drops 0.000M/s >>> Iter 2 (-10.774us): hits 150.717M/s (150.717M/prod), drops 0.000M/s >>> Iter 3 ( 3.807us): hits 151.435M/s (151.435M/prod), drops 0.000M/s >>> Summary: hits 150.488 ± 1.079M/s (150.488M/prod), drops 0.000 ± 0.000M/s >>> >>> $ ./bench -a -w1 -d2 -p4 count-global >>> Setting up benchmark 'count-global'... >>> Benchmark 'count-global' started. >>> Iter 0 ( 60.659us): hits 53.910M/s ( 13.477M/prod), drops 0.000M/s >>> Iter 1 (-17.658us): hits 53.722M/s ( 13.431M/prod), drops 0.000M/s >>> Iter 2 ( 5.865us): hits 53.495M/s ( 13.374M/prod), drops 0.000M/s >>> Iter 3 ( 0.104us): hits 53.606M/s ( 13.402M/prod), drops 0.000M/s >>> Summary: hits 53.608 ± 0.113M/s ( 13.402M/prod), drops 0.000 ± 0.000M/s >>> >>> $ ./bench -a -w1 -d2 -p4 count-local >>> Setting up benchmark 'count-local'... >>> Benchmark 'count-local' started. >>> Iter 0 ( 23.388us): hits 640.450M/s (160.113M/prod), drops 0.000M/s >>> Iter 1 ( 2.291us): hits 605.661M/s (151.415M/prod), drops 0.000M/s >>> Iter 2 ( -6.415us): hits 607.092M/s (151.773M/prod), drops 0.000M/s >>> Iter 3 ( -1.361us): hits 601.796M/s (150.449M/prod), drops 0.000M/s >>> Summary: hits 604.849 ± 2.739M/s (151.212M/prod), drops 0.000 ± 0.000M/s >>> >>> Signed-off-by: Andrii Nakryiko <andriin@fb.com> >>> --- >>> tools/testing/selftests/bpf/.gitignore | 1 + >>> tools/testing/selftests/bpf/Makefile | 13 +- >>> tools/testing/selftests/bpf/bench.c | 372 ++++++++++++++++++ >>> tools/testing/selftests/bpf/bench.h | 74 ++++ >>> .../selftests/bpf/benchs/bench_count.c | 91 +++++ >>> 5 files changed, 550 insertions(+), 1 deletion(-) >>> create mode 100644 tools/testing/selftests/bpf/bench.c >>> create mode 100644 tools/testing/selftests/bpf/bench.h >>> create mode 100644 tools/testing/selftests/bpf/benchs/bench_count.c >>> > > [...] > > trimming is good :) > >>> + >>> +void hits_drops_report_final(struct bench_res res[], int res_cnt) >>> +{ >>> + int i; >>> + double hits_mean = 0.0, drops_mean = 0.0; >>> + double hits_stddev = 0.0, drops_stddev = 0.0; >>> + >>> + for (i = 0; i < res_cnt; i++) { >>> + hits_mean += res[i].hits / 1000000.0 / (0.0 + res_cnt); >>> + drops_mean += res[i].drops / 1000000.0 / (0.0 + res_cnt); >>> + } >>> + >>> + if (res_cnt > 1) { >>> + for (i = 0; i < res_cnt; i++) { >>> + hits_stddev += (hits_mean - res[i].hits / 1000000.0) * >>> + (hits_mean - res[i].hits / 1000000.0) / >>> + (res_cnt - 1.0); >>> + drops_stddev += (drops_mean - res[i].drops / 1000000.0) * >>> + (drops_mean - res[i].drops / 1000000.0) / >>> + (res_cnt - 1.0); >>> + } >>> + hits_stddev = sqrt(hits_stddev); >>> + drops_stddev = sqrt(drops_stddev); >>> + } >>> + printf("Summary: hits %8.3lf \u00B1 %5.3lfM/s (%7.3lfM/prod), ", >>> + hits_mean, hits_stddev, hits_mean / env.producer_cnt); >>> + printf("drops %8.3lf \u00B1 %5.3lfM/s\n", >>> + drops_mean, drops_stddev); >> >> The unicode char \u00B1 (for +|-) may cause some old compiler (e.g., >> 4.8.5) warnings as it needs C99 mode. >> >> /data/users/yhs/work/net-next/tools/testing/selftests/bpf/bench.c:91:9: >> warning: universal character names are only valid in C++ and C99 >> [enabled by default] >> printf("Summary: hits %8.3lf \u00B1 %5.3lfM/s (%7.3lfM/prod), ", >> >> "+|-" is alternative, but not as good as \u00B1 indeed. Newer >> compiler may already have the default C99. Maybe we can just add >> C99 for build `bench`? > > I think I'm fine with ancient compiler emitting harmless warning for > code under selftests/bpf, honestly... > >> >>> +} >>> + >>> +const char *argp_program_version = "benchmark"; >>> +const char *argp_program_bug_address = "<bpf@vger.kernel.org>"; >>> +const char argp_program_doc[] = >>> +"benchmark Generic benchmarking framework.\n" >>> +"\n" >>> +"This tool runs benchmarks.\n" >>> +"\n" >>> +"USAGE: benchmark <bench-name>\n" >>> +"\n" >>> +"EXAMPLES:\n" >>> +" # run 'count-local' benchmark with 1 producer and 1 consumer\n" >>> +" benchmark count-local\n" >>> +" # run 'count-local' with 16 producer and 8 consumer thread, pinned to CPUs\n" >>> +" benchmark -p16 -c8 -a count-local\n"; >> >> Some of the above global variables probably are statics. >> But I do not have a strong preference on this. > > Oh, it's actually global variables argp library expects, they have to be global. > >> >>> + >>> +static const struct argp_option opts[] = { >>> + { "list", 'l', NULL, 0, "List available benchmarks"}, >>> + { "duration", 'd', "SEC", 0, "Duration of benchmark, seconds"}, >>> + { "warmup", 'w', "SEC", 0, "Warm-up period, seconds"}, >>> + { "producers", 'p', "NUM", 0, "Number of producer threads"}, >>> + { "consumers", 'c', "NUM", 0, "Number of consumer threads"}, >>> + { "verbose", 'v', NULL, 0, "Verbose debug output"}, >>> + { "affinity", 'a', NULL, 0, "Set consumer/producer thread affinity"}, >>> + { "b2b", 'b', NULL, 0, "Back-to-back mode"}, >>> + { "rb-output", 10001, NULL, 0, "Set consumer/producer thread affinity"}, >> >> I did not see b2b and rb-output options are processed in this file. > > Slipped through the rebasing cracks from the future ringbuf > benchmarks, will remove it. I also figured out a way to do this more > modular anyways (child parsers in argp). > >> >>> + {}, >>> +}; >>> + > > [...] > >>> + for (i = 0; i < env.consumer_cnt; i++) { >>> + err = pthread_create(&state.consumers[i], NULL, >>> + bench->consumer_thread, (void *)(long)i); >>> + if (err) { >>> + fprintf(stderr, "failed to create consumer thread #%d: %d\n", >>> + i, -errno); >>> + exit(1); >>> + } >>> + if (env.affinity) >>> + set_thread_affinity(state.consumers[i], i); >>> + } >>> + for (i = 0; i < env.producer_cnt; i++) { >>> + err = pthread_create(&state.producers[i], NULL, >>> + bench->producer_thread, (void *)(long)i); >>> + if (err) { >>> + fprintf(stderr, "failed to create producer thread #%d: %d\n", >>> + i, -errno); >>> + exit(1); >>> + } >>> + if (env.affinity) >>> + set_thread_affinity(state.producers[i], >>> + env.consumer_cnt + i); >> >> Here, we bind consumer threads first and then producer threads, I think >> this is probably just arbitrary choice? > > yep, almost arbitrary. Most of my cases have 1 consumer and >=1 > producers, so it was convenient to have consumer pinned to same CPU, > regardless of how many producers I have. > >> >> In certain cases, I think people may want to have more advanced binding >> scenarios, e.g., for hyperthreading, binding consumer and producer on >> the same core or different cores etc. One possibility is to introduce >> -c option similar to taskset. If -c not supplied, you can have >> the current default. Otherwise, using -c list. >> > > well, taskset's job is simpler, it takes a list of CPUs for single > PID, if I understand correctly. Here we have many threads, each might > have different CPU or even CPUs. But I agree that for some benchmarks > it's going to be critical to control this precisely. Here's how I'm > going to allows most flexibility without too much complexity. > > --prod-affinity 1,2,10-16,100 -- will specify a set of CPUs for > producers. First producer will use CPU with least index form that > list. Second will take second, and so on. If there are less CPUs > provided than necessary - it's an error. If more - it's fine. > > Then for consumers will add independent --cons-affinity parameters, > which will do the same for consumer threads. > > Having two independent lists will allow to test scenarios where we > want producers and consumers to fight for the same CPU. > > Does this sound ok? This sounds good, more than what I asked. > >>> + } >>> + >>> + printf("Benchmark '%s' started.\n", bench->name); >>> +} >>> + > > [...]
diff --git a/tools/testing/selftests/bpf/.gitignore b/tools/testing/selftests/bpf/.gitignore index 3ff031972975..1bb204cee853 100644 --- a/tools/testing/selftests/bpf/.gitignore +++ b/tools/testing/selftests/bpf/.gitignore @@ -38,3 +38,4 @@ test_cpp /bpf_gcc /tools /runqslower +/bench diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile index 3d942be23d09..289fffbf975e 100644 --- a/tools/testing/selftests/bpf/Makefile +++ b/tools/testing/selftests/bpf/Makefile @@ -77,7 +77,7 @@ TEST_PROGS_EXTENDED := with_addr.sh \ # Compile but not part of 'make run_tests' TEST_GEN_PROGS_EXTENDED = test_sock_addr test_skb_cgroup_id_user \ flow_dissector_load test_flow_dissector test_tcp_check_syncookie_user \ - test_lirc_mode2_user xdping test_cpp runqslower + test_lirc_mode2_user xdping test_cpp runqslower bench TEST_CUSTOM_PROGS = urandom_read @@ -405,6 +405,17 @@ $(OUTPUT)/test_cpp: test_cpp.cpp $(OUTPUT)/test_core_extern.skel.h $(BPFOBJ) $(call msg,CXX,,$@) $(CXX) $(CFLAGS) $^ $(LDLIBS) -o $@ +# Benchmark runner +$(OUTPUT)/bench_%.o: benchs/bench_%.c bench.h + $(call msg,CC,,$@) + $(CC) $(CFLAGS) -c $(filter %.c,$^) $(LDLIBS) -o $@ +$(OUTPUT)/bench.o: bench.h +$(OUTPUT)/bench: LDLIBS += -lm +$(OUTPUT)/bench: $(OUTPUT)/bench.o \ + $(OUTPUT)/bench_count.o + $(call msg,BINARY,,$@) + $(CC) $(LDFLAGS) -o $@ $(filter %.a %.o,$^) $(LDLIBS) + EXTRA_CLEAN := $(TEST_CUSTOM_PROGS) $(SCRATCH_DIR) \ prog_tests/tests.h map_tests/tests.h verifier/tests.h \ feature \ diff --git a/tools/testing/selftests/bpf/bench.c b/tools/testing/selftests/bpf/bench.c new file mode 100644 index 000000000000..dddc97cd4db6 --- /dev/null +++ b/tools/testing/selftests/bpf/bench.c @@ -0,0 +1,372 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2020 Facebook */ +#define _GNU_SOURCE +#include <argp.h> +#include <linux/compiler.h> +#include <sys/time.h> +#include <sched.h> +#include <fcntl.h> +#include <pthread.h> +#include <sys/sysinfo.h> +#include <sys/resource.h> +#include <signal.h> +#include "bench.h" + +struct env env = { + .warmup_sec = 1, + .duration_sec = 5, + .affinity = false, + .consumer_cnt = 1, + .producer_cnt = 1, +}; + +static int libbpf_print_fn(enum libbpf_print_level level, + const char *format, va_list args) +{ + if (level == LIBBPF_DEBUG && !env.verbose) + return 0; + return vfprintf(stderr, format, args); +} + +static int bump_memlock_rlimit(void) +{ + struct rlimit rlim_new = { + .rlim_cur = RLIM_INFINITY, + .rlim_max = RLIM_INFINITY, + }; + + return setrlimit(RLIMIT_MEMLOCK, &rlim_new); +} + +void setup_libbpf() +{ + int err; + + libbpf_set_print(libbpf_print_fn); + + err = bump_memlock_rlimit(); + if (err) + fprintf(stderr, "failed to increase RLIMIT_MEMLOCK: %d", err); +} + +void hits_drops_report_progress(int iter, struct bench_res *res, long delta_ns) +{ + double hits_per_sec, drops_per_sec; + double hits_per_prod; + + hits_per_sec = res->hits / 1000000.0 / (delta_ns / 1000000000.0); + hits_per_prod = hits_per_sec / env.producer_cnt; + drops_per_sec = res->drops / 1000000.0 / (delta_ns / 1000000000.0); + + printf("Iter %3d (%7.3lfus): ", + iter, (delta_ns - 1000000000) / 1000.0); + + printf("hits %8.3lfM/s (%7.3lfM/prod), drops %8.3lfM/s\n", + hits_per_sec, hits_per_prod, drops_per_sec); +} + +void hits_drops_report_final(struct bench_res res[], int res_cnt) +{ + int i; + double hits_mean = 0.0, drops_mean = 0.0; + double hits_stddev = 0.0, drops_stddev = 0.0; + + for (i = 0; i < res_cnt; i++) { + hits_mean += res[i].hits / 1000000.0 / (0.0 + res_cnt); + drops_mean += res[i].drops / 1000000.0 / (0.0 + res_cnt); + } + + if (res_cnt > 1) { + for (i = 0; i < res_cnt; i++) { + hits_stddev += (hits_mean - res[i].hits / 1000000.0) * + (hits_mean - res[i].hits / 1000000.0) / + (res_cnt - 1.0); + drops_stddev += (drops_mean - res[i].drops / 1000000.0) * + (drops_mean - res[i].drops / 1000000.0) / + (res_cnt - 1.0); + } + hits_stddev = sqrt(hits_stddev); + drops_stddev = sqrt(drops_stddev); + } + printf("Summary: hits %8.3lf \u00B1 %5.3lfM/s (%7.3lfM/prod), ", + hits_mean, hits_stddev, hits_mean / env.producer_cnt); + printf("drops %8.3lf \u00B1 %5.3lfM/s\n", + drops_mean, drops_stddev); +} + +const char *argp_program_version = "benchmark"; +const char *argp_program_bug_address = "<bpf@vger.kernel.org>"; +const char argp_program_doc[] = +"benchmark Generic benchmarking framework.\n" +"\n" +"This tool runs benchmarks.\n" +"\n" +"USAGE: benchmark <bench-name>\n" +"\n" +"EXAMPLES:\n" +" # run 'count-local' benchmark with 1 producer and 1 consumer\n" +" benchmark count-local\n" +" # run 'count-local' with 16 producer and 8 consumer thread, pinned to CPUs\n" +" benchmark -p16 -c8 -a count-local\n"; + +static const struct argp_option opts[] = { + { "list", 'l', NULL, 0, "List available benchmarks"}, + { "duration", 'd', "SEC", 0, "Duration of benchmark, seconds"}, + { "warmup", 'w', "SEC", 0, "Warm-up period, seconds"}, + { "producers", 'p', "NUM", 0, "Number of producer threads"}, + { "consumers", 'c', "NUM", 0, "Number of consumer threads"}, + { "verbose", 'v', NULL, 0, "Verbose debug output"}, + { "affinity", 'a', NULL, 0, "Set consumer/producer thread affinity"}, + { "b2b", 'b', NULL, 0, "Back-to-back mode"}, + { "rb-output", 10001, NULL, 0, "Set consumer/producer thread affinity"}, + {}, +}; + +static error_t parse_arg(int key, char *arg, struct argp_state *state) +{ + static int pos_args; + + switch (key) { + case 'v': + env.verbose = true; + break; + case 'l': + env.list = true; + break; + case 'd': + env.duration_sec = strtol(arg, NULL, 10); + if (env.duration_sec <= 0) { + fprintf(stderr, "Invalid duration: %s\n", arg); + argp_usage(state); + } + break; + case 'w': + env.warmup_sec = strtol(arg, NULL, 10); + if (env.warmup_sec <= 0) { + fprintf(stderr, "Invalid warm-up duration: %s\n", arg); + argp_usage(state); + } + break; + case 'p': + env.producer_cnt = strtol(arg, NULL, 10); + if (env.producer_cnt <= 0) { + fprintf(stderr, "Invalid producer count: %s\n", arg); + argp_usage(state); + } + break; + case 'c': + env.consumer_cnt = strtol(arg, NULL, 10); + if (env.consumer_cnt <= 0) { + fprintf(stderr, "Invalid consumer count: %s\n", arg); + argp_usage(state); + } + break; + case 'a': + env.affinity = true; + break; + case ARGP_KEY_ARG: + if (pos_args++) { + fprintf(stderr, + "Unrecognized positional argument: %s\n", arg); + argp_usage(state); + } + env.bench_name = strdup(arg); + break; + default: + return ARGP_ERR_UNKNOWN; + } + return 0; +} + +static void parse_cmdline_args(int argc, char **argv) +{ + static const struct argp argp = { + .options = opts, + .parser = parse_arg, + .doc = argp_program_doc, + }; + if (argp_parse(&argp, argc, argv, 0, NULL, NULL)) + exit(1); + if (!env.list && !env.bench_name) { + argp_help(&argp, stderr, ARGP_HELP_DOC, "bench"); + exit(1); + } +} + +static void collect_measurements(long delta_ns); + +static __u64 last_time_ns; +static void sigalarm_handler(int signo) +{ + long new_time_ns = get_time_ns(); + long delta_ns = new_time_ns - last_time_ns; + + collect_measurements(delta_ns); + + last_time_ns = new_time_ns; +} + +/* set up periodic 1-second timer */ +static void setup_timer() +{ + static struct sigaction sigalarm_action = { + .sa_handler = sigalarm_handler, + }; + struct itimerval timer_settings = {}; + int err; + + last_time_ns = get_time_ns(); + err = sigaction(SIGALRM, &sigalarm_action, NULL); + if (err < 0) { + fprintf(stderr, "failed to install SIGALARM handler: %d\n", -errno); + exit(1); + } + timer_settings.it_interval.tv_sec = 1; + timer_settings.it_value.tv_sec = 1; + err = setitimer(ITIMER_REAL, &timer_settings, NULL); + if (err < 0) { + fprintf(stderr, "failed to arm interval timer: %d\n", -errno); + exit(1); + } +} + +static void set_thread_affinity(pthread_t thread, int cpu) +{ + cpu_set_t cpuset; + + CPU_ZERO(&cpuset); + CPU_SET(cpu, &cpuset); + if (pthread_setaffinity_np(thread, sizeof(cpuset), &cpuset)) { + fprintf(stderr, "setting affinity to CPU #%d failed: %d\n", + cpu, errno); + exit(1); + } +} + +static struct bench_state { + int res_cnt; + struct bench_res *results; + pthread_t *consumers; + pthread_t *producers; +} state; + +const struct bench *bench = NULL; + +extern const struct bench bench_count_global; +extern const struct bench bench_count_local; + +static const struct bench *benchs[] = { + &bench_count_global, + &bench_count_local, +}; + +static void setup_benchmark() +{ + int i, err; + + if (!env.bench_name) { + fprintf(stderr, "benchmark name is not specified\n"); + exit(1); + } + + for (i = 0; i < ARRAY_SIZE(benchs); i++) { + if (strcmp(benchs[i]->name, env.bench_name) == 0) { + bench = benchs[i]; + break; + } + } + if (!bench) { + fprintf(stderr, "benchmark '%s' not found\n", env.bench_name); + exit(1); + } + + printf("Setting up benchmark '%s'...\n", bench->name); + + state.producers = calloc(env.producer_cnt, sizeof(*state.producers)); + state.consumers = calloc(env.consumer_cnt, sizeof(*state.consumers)); + state.results = calloc(env.duration_sec + env.warmup_sec + 2, + sizeof(*state.results)); + if (!state.producers || !state.consumers || !state.results) + exit(1); + + if (bench->validate) + bench->validate(); + if (bench->setup) + bench->setup(); + + for (i = 0; i < env.consumer_cnt; i++) { + err = pthread_create(&state.consumers[i], NULL, + bench->consumer_thread, (void *)(long)i); + if (err) { + fprintf(stderr, "failed to create consumer thread #%d: %d\n", + i, -errno); + exit(1); + } + if (env.affinity) + set_thread_affinity(state.consumers[i], i); + } + for (i = 0; i < env.producer_cnt; i++) { + err = pthread_create(&state.producers[i], NULL, + bench->producer_thread, (void *)(long)i); + if (err) { + fprintf(stderr, "failed to create producer thread #%d: %d\n", + i, -errno); + exit(1); + } + if (env.affinity) + set_thread_affinity(state.producers[i], + env.consumer_cnt + i); + } + + printf("Benchmark '%s' started.\n", bench->name); +} + +static pthread_mutex_t bench_done_mtx = PTHREAD_MUTEX_INITIALIZER; +static pthread_cond_t bench_done = PTHREAD_COND_INITIALIZER; + +static void collect_measurements(long delta_ns) { + int iter = state.res_cnt++; + struct bench_res *res = &state.results[iter]; + + bench->measure(res); + + if (bench->report_progress) + bench->report_progress(iter, res, delta_ns); + + if (iter == env.duration_sec + env.warmup_sec) { + pthread_mutex_lock(&bench_done_mtx); + pthread_cond_signal(&bench_done); + pthread_mutex_unlock(&bench_done_mtx); + } +} + +int main(int argc, char **argv) +{ + parse_cmdline_args(argc, argv); + + if (env.list) { + int i; + + printf("Available benchmarks:\n"); + for (i = 0; i < ARRAY_SIZE(benchs); i++) { + printf("- %s\n", benchs[i]->name); + } + return 0; + } + + setup_benchmark(); + + setup_timer(); + + pthread_mutex_lock(&bench_done_mtx); + pthread_cond_wait(&bench_done, &bench_done_mtx); + pthread_mutex_unlock(&bench_done_mtx); + + if (bench->report_final) + /* skip first sample */ + bench->report_final(state.results + env.warmup_sec, + state.res_cnt - env.warmup_sec); + + return 0; +} + diff --git a/tools/testing/selftests/bpf/bench.h b/tools/testing/selftests/bpf/bench.h new file mode 100644 index 000000000000..08aa0c5b1177 --- /dev/null +++ b/tools/testing/selftests/bpf/bench.h @@ -0,0 +1,74 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#pragma once +#include <stdlib.h> +#include <stdbool.h> +#include <linux/err.h> +#include <errno.h> +#include <unistd.h> +#include <bpf/bpf.h> +#include <bpf/libbpf.h> +#include <math.h> +#include <time.h> +#include <sys/syscall.h> + +struct env { + char *bench_name; + int duration_sec; + int warmup_sec; + bool verbose; + bool list; + bool back2back; + bool affinity; + int consumer_cnt; + int producer_cnt; +}; + +struct bench_res { + long hits; + long drops; +}; + +struct bench { + const char *name; + void (*validate)(); + void (*setup)(); + void *(*producer_thread)(void *ctx); + void *(*consumer_thread)(void *ctx); + void (*measure)(struct bench_res* res); + void (*report_progress)(int iter, struct bench_res* res, long delta_ns); + void (*report_final)(struct bench_res res[], int res_cnt); +}; + +struct counter { + long value; +} __attribute__((aligned(128))); + +extern struct env env; +extern const struct bench *bench; + +void setup_libbpf(); +void hits_drops_report_progress(int iter, struct bench_res *res, long delta_ns); +void hits_drops_report_final(struct bench_res res[], int res_cnt); + +static inline __u64 get_time_ns() { + struct timespec t; + + clock_gettime(CLOCK_MONOTONIC, &t); + + return (u64)t.tv_sec * 1000000000 + t.tv_nsec; +} + +static inline void atomic_inc(long *value) +{ + (void)__atomic_add_fetch(value, 1, __ATOMIC_RELAXED); +} + +static inline void atomic_add(long *value, long n) +{ + (void)__atomic_add_fetch(value, n, __ATOMIC_RELAXED); +} + +static inline long atomic_swap(long *value, long n) +{ + return __atomic_exchange_n(value, n, __ATOMIC_RELAXED); +} diff --git a/tools/testing/selftests/bpf/benchs/bench_count.c b/tools/testing/selftests/bpf/benchs/bench_count.c new file mode 100644 index 000000000000..befba7a82643 --- /dev/null +++ b/tools/testing/selftests/bpf/benchs/bench_count.c @@ -0,0 +1,91 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2020 Facebook */ +#include "bench.h" + +/* COUNT-GLOBAL benchmark */ + +static struct count_global_ctx { + struct counter hits; +} count_global_ctx; + +static void *count_global_producer(void *input) +{ + struct count_global_ctx *ctx = &count_global_ctx; + + while (true) { + atomic_inc(&ctx->hits.value); + } + return NULL; +} + +static void *count_global_consumer(void *input) +{ + return NULL; +} + +static void count_global_measure(struct bench_res *res) +{ + struct count_global_ctx *ctx = &count_global_ctx; + + res->hits = atomic_swap(&ctx->hits.value, 0); +} + +/* COUNT-local benchmark */ + +static struct count_local_ctx { + struct counter *hits; +} count_local_ctx; + +static void count_local_setup() +{ + struct count_local_ctx *ctx = &count_local_ctx; + + ctx->hits = calloc(env.consumer_cnt, sizeof(*ctx->hits)); + if (!ctx->hits) + exit(1); +} + +static void *count_local_producer(void *input) +{ + struct count_local_ctx *ctx = &count_local_ctx; + int idx = (long)input; + + while (true) { + atomic_inc(&ctx->hits[idx].value); + } + return NULL; +} + +static void *count_local_consumer(void *input) +{ + return NULL; +} + +static void count_local_measure(struct bench_res *res) +{ + struct count_local_ctx *ctx = &count_local_ctx; + int i; + + for (i = 0; i < env.producer_cnt; i++) { + res->hits += atomic_swap(&ctx->hits[i].value, 0); + } +} + +const struct bench bench_count_global = { + .name = "count-global", + .producer_thread = count_global_producer, + .consumer_thread = count_global_consumer, + .measure = count_global_measure, + .report_progress = hits_drops_report_progress, + .report_final = hits_drops_report_final, +}; + +const struct bench bench_count_local = { + .name = "count-local", + .setup = count_local_setup, + .producer_thread = count_local_producer, + .consumer_thread = count_local_consumer, + .measure = count_local_measure, + .report_progress = hits_drops_report_progress, + .report_final = hits_drops_report_final, +};
While working on BPF ringbuf implementation, testing, and benchmarking, I've developed a pretty generic and modular benchmark runner, which seems to be generically useful, as I've already used it for one more purpose (testing fastest way to trigger BPF program, to minimize overhead of in-kernel code). This patch adds generic part of benchmark runner and sets up Makefile for extending it with more sets of benchmarks. Benchmarker itself operates by spinning up specified number of producer and consumer threads, setting up interval timer sending SIGALARM signal to application once a second. Every second, current snapshot with hits/drops counters are collected and stored in an array. Drops are useful for producer/consumer benchmarks in which producer might overwhelm consumers. Once test finishes after given amount of warm-up and testing seconds, mean and stddev are calculated (ignoring warm-up results) and is printed out to stdout. This setup seems to give consistent and accurate results. To validate behavior, I added two atomic counting tests: global and local. For global one, all the producer threads are atomically incrementing same counter as fast as possible. This, of course, leads to huge drop of performance once there is more than one producer thread due to CPUs fighting for the same memory location. Local counting, on the other hand, maintains one counter per each producer thread, incremented independently. Once per second, all counters are read and added together to form final "counting throughput" measurement. As expected, such setup demonstrates linear scalability with number of producers (as long as there are enough physical CPU cores, of course). See example output below. Also, this setup can nicely demonstrate disastrous effects of false sharing, if care is not taken to take those per-producer counters apart into independent cache lines. Demo output shows global counter first with 1 producer, then with 4. Both total and per-producer performance significantly drop. The last run is local counter with 4 producers, demonstrating near-perfect scalability. $ ./bench -a -w1 -d2 -p1 count-global Setting up benchmark 'count-global'... Benchmark 'count-global' started. Iter 0 ( 24.822us): hits 148.179M/s (148.179M/prod), drops 0.000M/s Iter 1 ( 37.939us): hits 149.308M/s (149.308M/prod), drops 0.000M/s Iter 2 (-10.774us): hits 150.717M/s (150.717M/prod), drops 0.000M/s Iter 3 ( 3.807us): hits 151.435M/s (151.435M/prod), drops 0.000M/s Summary: hits 150.488 ± 1.079M/s (150.488M/prod), drops 0.000 ± 0.000M/s $ ./bench -a -w1 -d2 -p4 count-global Setting up benchmark 'count-global'... Benchmark 'count-global' started. Iter 0 ( 60.659us): hits 53.910M/s ( 13.477M/prod), drops 0.000M/s Iter 1 (-17.658us): hits 53.722M/s ( 13.431M/prod), drops 0.000M/s Iter 2 ( 5.865us): hits 53.495M/s ( 13.374M/prod), drops 0.000M/s Iter 3 ( 0.104us): hits 53.606M/s ( 13.402M/prod), drops 0.000M/s Summary: hits 53.608 ± 0.113M/s ( 13.402M/prod), drops 0.000 ± 0.000M/s $ ./bench -a -w1 -d2 -p4 count-local Setting up benchmark 'count-local'... Benchmark 'count-local' started. Iter 0 ( 23.388us): hits 640.450M/s (160.113M/prod), drops 0.000M/s Iter 1 ( 2.291us): hits 605.661M/s (151.415M/prod), drops 0.000M/s Iter 2 ( -6.415us): hits 607.092M/s (151.773M/prod), drops 0.000M/s Iter 3 ( -1.361us): hits 601.796M/s (150.449M/prod), drops 0.000M/s Summary: hits 604.849 ± 2.739M/s (151.212M/prod), drops 0.000 ± 0.000M/s Signed-off-by: Andrii Nakryiko <andriin@fb.com> --- tools/testing/selftests/bpf/.gitignore | 1 + tools/testing/selftests/bpf/Makefile | 13 +- tools/testing/selftests/bpf/bench.c | 372 ++++++++++++++++++ tools/testing/selftests/bpf/bench.h | 74 ++++ .../selftests/bpf/benchs/bench_count.c | 91 +++++ 5 files changed, 550 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/bpf/bench.c create mode 100644 tools/testing/selftests/bpf/bench.h create mode 100644 tools/testing/selftests/bpf/benchs/bench_count.c