| Message ID | 20190626061235.602633-2-andriin@fb.com |
|---|---|
| State | Changes Requested |
| Delegated to: | BPF Maintainers |
| Headers | show |
| Series | libbpf: add perf buffer abstraction and API | expand |
> On Jun 25, 2019, at 11:12 PM, Andrii Nakryiko <andriin@fb.com> wrote: > > BPF_MAP_TYPE_PERF_EVENT_ARRAY map is often used to send data from BPF program > to user space for additional processing. libbpf already has very low-level API > to read single CPU perf buffer, bpf_perf_event_read_simple(), but it's hard to > use and requires a lot of code to set everything up. This patch adds > perf_buffer abstraction on top of it, abstracting setting up and polling > per-CPU logic into simple and convenient API, similar to what BCC provides. > > perf_buffer__new() sets up per-CPU ring buffers and updates corresponding BPF > map entries. It accepts two user-provided callbacks: one for handling raw > samples and one for get notifications of lost samples due to buffer overflow. > > perf_buffer__poll() is used to fetch ring buffer data across all CPUs, > utilizing epoll instance. > > perf_buffer__free() does corresponding clean up and unsets FDs from BPF map. > > All APIs are not thread-safe. User should ensure proper locking/coordination if > used in multi-threaded set up. > > Signed-off-by: Andrii Nakryiko <andriin@fb.com> Acked-by: Song Liu <songliubraving@fb.com> > --- > tools/lib/bpf/libbpf.c | 282 +++++++++++++++++++++++++++++++++++++++ > tools/lib/bpf/libbpf.h | 12 ++ > tools/lib/bpf/libbpf.map | 5 +- > 3 files changed, 298 insertions(+), 1 deletion(-)
On 06/26/2019 08:12 AM, Andrii Nakryiko wrote: > BPF_MAP_TYPE_PERF_EVENT_ARRAY map is often used to send data from BPF program > to user space for additional processing. libbpf already has very low-level API > to read single CPU perf buffer, bpf_perf_event_read_simple(), but it's hard to > use and requires a lot of code to set everything up. This patch adds > perf_buffer abstraction on top of it, abstracting setting up and polling > per-CPU logic into simple and convenient API, similar to what BCC provides. > > perf_buffer__new() sets up per-CPU ring buffers and updates corresponding BPF > map entries. It accepts two user-provided callbacks: one for handling raw > samples and one for get notifications of lost samples due to buffer overflow. > > perf_buffer__poll() is used to fetch ring buffer data across all CPUs, > utilizing epoll instance. > > perf_buffer__free() does corresponding clean up and unsets FDs from BPF map. > > All APIs are not thread-safe. User should ensure proper locking/coordination if > used in multi-threaded set up. > > Signed-off-by: Andrii Nakryiko <andriin@fb.com> Aside from current feedback, this series generally looks great! Two small things: > diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map > index 2382fbda4cbb..10f48103110a 100644 > --- a/tools/lib/bpf/libbpf.map > +++ b/tools/lib/bpf/libbpf.map > @@ -170,13 +170,16 @@ LIBBPF_0.0.4 { > btf_dump__dump_type; > btf_dump__free; > btf_dump__new; > - btf__parse_elf; > bpf_object__load_xattr; > bpf_program__attach_kprobe; > bpf_program__attach_perf_event; > bpf_program__attach_raw_tracepoint; > bpf_program__attach_tracepoint; > bpf_program__attach_uprobe; > + btf__parse_elf; > libbpf_num_possible_cpus; > libbpf_perf_event_disable_and_close; > + perf_buffer__free; > + perf_buffer__new; > + perf_buffer__poll; We should prefix with libbpf_* given it's not strictly BPF-only and rather helper function. Also, we should convert bpftool (tools/bpf/bpftool/map_perf_ring.c) to make use of these new helpers instead of open-coding there. Thanks, Daniel
On Thu, Jun 27, 2019 at 2:04 PM Daniel Borkmann <daniel@iogearbox.net> wrote: > > On 06/26/2019 08:12 AM, Andrii Nakryiko wrote: > > BPF_MAP_TYPE_PERF_EVENT_ARRAY map is often used to send data from BPF program > > to user space for additional processing. libbpf already has very low-level API > > to read single CPU perf buffer, bpf_perf_event_read_simple(), but it's hard to > > use and requires a lot of code to set everything up. This patch adds > > perf_buffer abstraction on top of it, abstracting setting up and polling > > per-CPU logic into simple and convenient API, similar to what BCC provides. > > > > perf_buffer__new() sets up per-CPU ring buffers and updates corresponding BPF > > map entries. It accepts two user-provided callbacks: one for handling raw > > samples and one for get notifications of lost samples due to buffer overflow. > > > > perf_buffer__poll() is used to fetch ring buffer data across all CPUs, > > utilizing epoll instance. > > > > perf_buffer__free() does corresponding clean up and unsets FDs from BPF map. > > > > All APIs are not thread-safe. User should ensure proper locking/coordination if > > used in multi-threaded set up. > > > > Signed-off-by: Andrii Nakryiko <andriin@fb.com> > > Aside from current feedback, this series generally looks great! Two small things: > > > diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map > > index 2382fbda4cbb..10f48103110a 100644 > > --- a/tools/lib/bpf/libbpf.map > > +++ b/tools/lib/bpf/libbpf.map > > @@ -170,13 +170,16 @@ LIBBPF_0.0.4 { > > btf_dump__dump_type; > > btf_dump__free; > > btf_dump__new; > > - btf__parse_elf; > > bpf_object__load_xattr; > > bpf_program__attach_kprobe; > > bpf_program__attach_perf_event; > > bpf_program__attach_raw_tracepoint; > > bpf_program__attach_tracepoint; > > bpf_program__attach_uprobe; > > + btf__parse_elf; > > libbpf_num_possible_cpus; > > libbpf_perf_event_disable_and_close; > > + perf_buffer__free; > > + perf_buffer__new; > > + perf_buffer__poll; > > We should prefix with libbpf_* given it's not strictly BPF-only and rather > helper function. Well, perf_buffer is an object similar to `struct btf`, `struct bpf_program`, etc. So it seems appropriate to follow this "<object>__<method>" convention. Also, `struct libbpf_perf_buffer` and `libbpf_perf_buffer__new` looks verbose and pretty ugly, IMO. > > Also, we should convert bpftool (tools/bpf/bpftool/map_perf_ring.c) to make > use of these new helpers instead of open-coding there. Yep, absolutely, will do that as well, thanks for pointing me there! > > Thanks, > Daniel
On Thu, Jun 27, 2019 at 2:45 PM Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote: > > On Thu, Jun 27, 2019 at 2:04 PM Daniel Borkmann <daniel@iogearbox.net> wrote: > > > > On 06/26/2019 08:12 AM, Andrii Nakryiko wrote: > > > BPF_MAP_TYPE_PERF_EVENT_ARRAY map is often used to send data from BPF program > > > to user space for additional processing. libbpf already has very low-level API > > > to read single CPU perf buffer, bpf_perf_event_read_simple(), but it's hard to > > > use and requires a lot of code to set everything up. This patch adds > > > perf_buffer abstraction on top of it, abstracting setting up and polling > > > per-CPU logic into simple and convenient API, similar to what BCC provides. > > > > > > perf_buffer__new() sets up per-CPU ring buffers and updates corresponding BPF > > > map entries. It accepts two user-provided callbacks: one for handling raw > > > samples and one for get notifications of lost samples due to buffer overflow. > > > > > > perf_buffer__poll() is used to fetch ring buffer data across all CPUs, > > > utilizing epoll instance. > > > > > > perf_buffer__free() does corresponding clean up and unsets FDs from BPF map. > > > > > > All APIs are not thread-safe. User should ensure proper locking/coordination if > > > used in multi-threaded set up. > > > > > > Signed-off-by: Andrii Nakryiko <andriin@fb.com> > > > > Aside from current feedback, this series generally looks great! Two small things: > > > > > diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map > > > index 2382fbda4cbb..10f48103110a 100644 > > > --- a/tools/lib/bpf/libbpf.map > > > +++ b/tools/lib/bpf/libbpf.map > > > @@ -170,13 +170,16 @@ LIBBPF_0.0.4 { > > > btf_dump__dump_type; > > > btf_dump__free; > > > btf_dump__new; > > > - btf__parse_elf; > > > bpf_object__load_xattr; > > > bpf_program__attach_kprobe; > > > bpf_program__attach_perf_event; > > > bpf_program__attach_raw_tracepoint; > > > bpf_program__attach_tracepoint; > > > bpf_program__attach_uprobe; > > > + btf__parse_elf; > > > libbpf_num_possible_cpus; > > > libbpf_perf_event_disable_and_close; > > > + perf_buffer__free; > > > + perf_buffer__new; > > > + perf_buffer__poll; > > > > We should prefix with libbpf_* given it's not strictly BPF-only and rather > > helper function. > > Well, perf_buffer is an object similar to `struct btf`, `struct > bpf_program`, etc. So it seems appropriate to follow this > "<object>__<method>" convention. Also, `struct libbpf_perf_buffer` and > `libbpf_perf_buffer__new` looks verbose and pretty ugly, IMO. > > > > > Also, we should convert bpftool (tools/bpf/bpftool/map_perf_ring.c) to make > > use of these new helpers instead of open-coding there. > > Yep, absolutely, will do that as well, thanks for pointing me there! This turned out to require much bigger changes, as bpftool needed way more low-level control over structure of events and attachment policies (custom cpu index and map key). So I ended up having two APIs: 1. simple common-case perf_buffer__new with 2 callbacks, that attaches to all CPUs (up to max_elements of map) 2. perf_buffer__new_raw, that allows to provide custom perf_event_attr, callback that accepts pointer to raw perf event and allows to specify any set of cpu/map keys. bpftool uses the latter one. Please see v3. > > > > > Thanks, > > Daniel
diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c index 9a4199b51300..c74cc535902a 100644 --- a/tools/lib/bpf/libbpf.c +++ b/tools/lib/bpf/libbpf.c @@ -32,7 +32,9 @@ #include <linux/limits.h> #include <linux/perf_event.h> #include <linux/ring_buffer.h> +#include <sys/epoll.h> #include <sys/ioctl.h> +#include <sys/mman.h> #include <sys/stat.h> #include <sys/types.h> #include <sys/vfs.h> @@ -4322,6 +4324,286 @@ bpf_perf_event_read_simple(void *mmap_mem, size_t mmap_size, size_t page_size, return ret; } +struct perf_cpu_buf { + int fd; + void *base; /* mmap()'ed memory */ + void *buf; /* for reconstructing segmented data */ + size_t buf_size; +}; + +struct perf_buffer { + perf_buffer_sample_fn sample_cb; + perf_buffer_lost_fn lost_cb; + void *ctx; /* passed into callbacks */ + + size_t page_size; + size_t mmap_size; + struct perf_cpu_buf **cpu_bufs; + struct epoll_event *events; + int cpu_cnt; + int epfd; /* perf event FD */ + int mapfd; /* BPF_MAP_TYPE_PERF_EVENT_ARRAY BPF map FD */ +}; + +static void perf_buffer__free_cpu_buf(struct perf_buffer *pb, + struct perf_cpu_buf *cpu_buf, int cpu) +{ + if (!cpu_buf) + return; + if (cpu_buf->base && + munmap(cpu_buf->base, pb->mmap_size + pb->page_size)) + pr_warning("failed to munmap cpu_buf #%d\n", cpu); + if (cpu_buf->fd >= 0) { + ioctl(cpu_buf->fd, PERF_EVENT_IOC_DISABLE, 0); + close(cpu_buf->fd); + } + free(cpu_buf->buf); + free(cpu_buf); +} + +void perf_buffer__free(struct perf_buffer *pb) +{ + int i; + + if (!pb) + return; + if (pb->cpu_bufs) { + for (i = 0; i < pb->cpu_cnt && pb->cpu_bufs[i]; i++) { + struct perf_cpu_buf *cpu_buf = pb->cpu_bufs[i]; + + bpf_map_delete_elem(pb->mapfd, &i); + perf_buffer__free_cpu_buf(pb, cpu_buf, i); + } + free(pb->cpu_bufs); + } + if (pb->epfd >= 0) + close(pb->epfd); + free(pb->events); + free(pb); +} + +static struct perf_cpu_buf *perf_buffer__open_cpu_buf(struct perf_buffer *pb, + int cpu) +{ + struct perf_event_attr attr = {}; + struct perf_cpu_buf *cpu_buf; + char msg[STRERR_BUFSIZE]; + int err; + + cpu_buf = calloc(1, sizeof(*cpu_buf)); + if (!cpu_buf) + return ERR_PTR(-ENOMEM); + + attr.config = PERF_COUNT_SW_BPF_OUTPUT; + attr.type = PERF_TYPE_SOFTWARE; + attr.sample_type = PERF_SAMPLE_RAW; + attr.sample_period = 1; + attr.wakeup_events = 1; + cpu_buf->fd = syscall(__NR_perf_event_open, &attr, -1 /* pid */, cpu, + -1, PERF_FLAG_FD_CLOEXEC); + if (cpu_buf->fd < 0) { + err = -errno; + pr_warning("failed to open perf buffer event on cpu #%d: %s\n", + cpu, libbpf_strerror_r(err, msg, sizeof(msg))); + goto error; + } + + cpu_buf->base = mmap(NULL, pb->mmap_size + pb->page_size, + PROT_READ | PROT_WRITE, MAP_SHARED, + cpu_buf->fd, 0); + if (cpu_buf->base == MAP_FAILED) { + cpu_buf->base = NULL; + err = -errno; + pr_warning("failed to mmap perf buffer on cpu #%d: %s\n", + cpu, libbpf_strerror_r(err, msg, sizeof(msg))); + goto error; + } + + if (ioctl(cpu_buf->fd, PERF_EVENT_IOC_ENABLE, 0) < 0) { + err = -errno; + pr_warning("failed to enable perf buffer event on cpu #%d: %s\n", + cpu, libbpf_strerror_r(err, msg, sizeof(msg))); + goto error; + } + + return cpu_buf; + +error: + perf_buffer__free_cpu_buf(pb, cpu_buf, cpu); + return (struct perf_cpu_buf *)ERR_PTR(err); +} + +struct perf_buffer *perf_buffer__new(struct bpf_map *map, size_t page_cnt, + perf_buffer_sample_fn sample_cb, + perf_buffer_lost_fn lost_cb, void *ctx) +{ + char msg[STRERR_BUFSIZE]; + struct perf_buffer *pb; + int err, cpu; + + if (bpf_map__def(map)->type != BPF_MAP_TYPE_PERF_EVENT_ARRAY) { + pr_warning("map '%s' should be BPF_MAP_TYPE_PERF_EVENT_ARRAY\n", + bpf_map__name(map)); + return ERR_PTR(-EINVAL); + } + if (bpf_map__fd(map) < 0) { + pr_warning("map '%s' doesn't have associated FD\n", + bpf_map__name(map)); + return ERR_PTR(-EINVAL); + } + if (page_cnt & (page_cnt - 1)) { + pr_warning("page count should be power of two, but is %zu\n", + page_cnt); + return ERR_PTR(-EINVAL); + } + + pb = calloc(1, sizeof(*pb)); + if (!pb) + return ERR_PTR(-ENOMEM); + + pb->sample_cb = sample_cb; + pb->lost_cb = lost_cb; + pb->ctx = ctx; + pb->page_size = getpagesize(); + pb->mmap_size = pb->page_size * page_cnt; + pb->mapfd = bpf_map__fd(map); + + pb->epfd = epoll_create1(EPOLL_CLOEXEC); + if (pb->epfd < 0) { + err = -errno; + pr_warning("failed to create epoll instance: %s\n", + libbpf_strerror_r(err, msg, sizeof(msg))); + goto error; + } + + pb->cpu_cnt = libbpf_num_possible_cpus(); + if (pb->cpu_cnt < 0) { + err = pb->cpu_cnt; + goto error; + } + pb->events = calloc(pb->cpu_cnt, sizeof(*pb->events)); + if (!pb->events) { + err = -ENOMEM; + pr_warning("failed to allocate events: out of memory\n"); + goto error; + } + pb->cpu_bufs = calloc(pb->cpu_cnt, sizeof(*pb->cpu_bufs)); + if (!pb->cpu_bufs) { + err = -ENOMEM; + pr_warning("failed to allocate buffers: out of memory\n"); + goto error; + } + + for (cpu = 0; cpu < pb->cpu_cnt; cpu++) { + struct perf_cpu_buf *cpu_buf; + + cpu_buf = perf_buffer__open_cpu_buf(pb, cpu); + if (IS_ERR(cpu_buf)) { + err = PTR_ERR(cpu_buf); + goto error; + } + + pb->cpu_bufs[cpu] = cpu_buf; + + err = bpf_map_update_elem(pb->mapfd, &cpu, &cpu_buf->fd, 0); + if (err) { + pr_warning("failed to set cpu #%d perf FD %d: %s\n", + cpu, cpu_buf->fd, + libbpf_strerror_r(err, msg, sizeof(msg))); + goto error; + } + + pb->events[cpu].events = EPOLLIN; + pb->events[cpu].data.ptr = cpu_buf; + if (epoll_ctl(pb->epfd, EPOLL_CTL_ADD, cpu_buf->fd, + &pb->events[cpu]) < 0) { + err = -errno; + pr_warning("failed to epoll_ctl cpu #%d perf FD %d: %s\n", + cpu, cpu_buf->fd, + libbpf_strerror_r(err, msg, sizeof(msg))); + goto error; + } + } + + return pb; + +error: + if (pb) + perf_buffer__free(pb); + return ERR_PTR(err); +} + +struct perf_sample_raw { + struct perf_event_header header; + uint32_t size; + char data[0]; +}; + +struct perf_sample_lost { + struct perf_event_header header; + uint64_t id; + uint64_t lost; + uint64_t sample_id; +}; + +static enum bpf_perf_event_ret +perf_buffer__process_record(struct perf_event_header *e, void *ctx) +{ + struct perf_buffer *pb = ctx; + void *data = e; + + switch (e->type) { + case PERF_RECORD_SAMPLE: { + struct perf_sample_raw *s = data; + + pb->sample_cb(pb->ctx, s->data, s->size); + break; + } + case PERF_RECORD_LOST: { + struct perf_sample_lost *s = data; + + if (pb->lost_cb) + pb->lost_cb(pb->ctx, s->lost); + break; + } + default: + pr_warning("unknown perf sample type %d\n", e->type); + return LIBBPF_PERF_EVENT_ERROR; + } + return LIBBPF_PERF_EVENT_CONT; +} + +static int perf_buffer__process_records(struct perf_buffer *pb, + struct perf_cpu_buf *cpu_buf) +{ + enum bpf_perf_event_ret ret; + + ret = bpf_perf_event_read_simple(cpu_buf->base, pb->mmap_size, + pb->page_size, &cpu_buf->buf, + &cpu_buf->buf_size, + perf_buffer__process_record, pb); + if (ret != LIBBPF_PERF_EVENT_CONT) + return ret; + return 0; +} + +int perf_buffer__poll(struct perf_buffer *pb, int timeout_ms) +{ + int cnt, err; + + cnt = epoll_wait(pb->epfd, pb->events, pb->cpu_cnt, timeout_ms); + for (int i = 0; i < cnt; i++) { + struct perf_cpu_buf *cpu_buf = pb->events[i].data.ptr; + + err = perf_buffer__process_records(pb, cpu_buf); + if (err) { + pr_warning("error while processing records: %d\n", err); + return err; + } + } + return cnt < 0 ? -errno : cnt; +} + struct bpf_prog_info_array_desc { int array_offset; /* e.g. offset of jited_prog_insns */ int count_offset; /* e.g. offset of jited_prog_len */ diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h index bf7020a565c6..3bfde1a475ce 100644 --- a/tools/lib/bpf/libbpf.h +++ b/tools/lib/bpf/libbpf.h @@ -354,6 +354,18 @@ LIBBPF_API int bpf_prog_load(const char *file, enum bpf_prog_type type, LIBBPF_API int bpf_set_link_xdp_fd(int ifindex, int fd, __u32 flags); LIBBPF_API int bpf_get_link_xdp_id(int ifindex, __u32 *prog_id, __u32 flags); +struct perf_buffer; +typedef void (*perf_buffer_sample_fn)(void *ctx, void *data, __u32 size); +typedef void (*perf_buffer_lost_fn)(void *ctx, __u64 cnt); + +LIBBPF_API struct perf_buffer *perf_buffer__new(struct bpf_map *map, + size_t page_cnt, + perf_buffer_sample_fn sample_cb, + perf_buffer_lost_fn lost_cb, + void *ctx); +LIBBPF_API void perf_buffer__free(struct perf_buffer *pb); +LIBBPF_API int perf_buffer__poll(struct perf_buffer *pb, int timeout_ms); + enum bpf_perf_event_ret { LIBBPF_PERF_EVENT_DONE = 0, LIBBPF_PERF_EVENT_ERROR = -1, diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map index 2382fbda4cbb..10f48103110a 100644 --- a/tools/lib/bpf/libbpf.map +++ b/tools/lib/bpf/libbpf.map @@ -170,13 +170,16 @@ LIBBPF_0.0.4 { btf_dump__dump_type; btf_dump__free; btf_dump__new; - btf__parse_elf; bpf_object__load_xattr; bpf_program__attach_kprobe; bpf_program__attach_perf_event; bpf_program__attach_raw_tracepoint; bpf_program__attach_tracepoint; bpf_program__attach_uprobe; + btf__parse_elf; libbpf_num_possible_cpus; libbpf_perf_event_disable_and_close; + perf_buffer__free; + perf_buffer__new; + perf_buffer__poll; } LIBBPF_0.0.3;
BPF_MAP_TYPE_PERF_EVENT_ARRAY map is often used to send data from BPF program to user space for additional processing. libbpf already has very low-level API to read single CPU perf buffer, bpf_perf_event_read_simple(), but it's hard to use and requires a lot of code to set everything up. This patch adds perf_buffer abstraction on top of it, abstracting setting up and polling per-CPU logic into simple and convenient API, similar to what BCC provides. perf_buffer__new() sets up per-CPU ring buffers and updates corresponding BPF map entries. It accepts two user-provided callbacks: one for handling raw samples and one for get notifications of lost samples due to buffer overflow. perf_buffer__poll() is used to fetch ring buffer data across all CPUs, utilizing epoll instance. perf_buffer__free() does corresponding clean up and unsets FDs from BPF map. All APIs are not thread-safe. User should ensure proper locking/coordination if used in multi-threaded set up. Signed-off-by: Andrii Nakryiko <andriin@fb.com> --- tools/lib/bpf/libbpf.c | 282 +++++++++++++++++++++++++++++++++++++++ tools/lib/bpf/libbpf.h | 12 ++ tools/lib/bpf/libbpf.map | 5 +- 3 files changed, 298 insertions(+), 1 deletion(-)