| Message ID | 20201105043402.2530976-1-andrii@kernel.org |
|---|---|
| Headers | show |
| Series | libbpf: split BTF support | expand |
On Wed, 4 Nov 2020 20:33:50 -0800 Andrii Nakryiko <andrii@kernel.org> wrote: > This patch set adds support for generating and deduplicating split BTF. This > is an enhancement to the BTF, which allows to designate one BTF as the "base > BTF" (e.g., vmlinux BTF), and one or more other BTFs as "split BTF" (e.g., > kernel module BTF), which are building upon and extending base BTF with extra > types and strings. > > Once loaded, split BTF appears as a single unified BTF superset of base BTF, > with continuous and transparent numbering scheme. This allows all the existing > users of BTF to work correctly and stay agnostic to the base/split BTFs > composition. The only difference is in how to instantiate split BTF: it > requires base BTF to be alread instantiated and passed to btf__new_xxx_split() > or btf__parse_xxx_split() "constructors" explicitly. > > This split approach is necessary if we are to have a reasonably-sized kernel > module BTFs. By deduping each kernel module's BTF individually, resulting > module BTFs contain copies of a lot of kernel types that are already present > in vmlinux BTF. Even those single copies result in a big BTF size bloat. On my > kernel configuration with 700 modules built, non-split BTF approach results in > 115MBs of BTFs across all modules. With split BTF deduplication approach, > total size is down to 5.2MBs total, which is on part with vmlinux BTF (at > around 4MBs). This seems reasonable and practical. As to why we'd need kernel > module BTFs, that should be pretty obvious to anyone using BPF at this point, > as it allows all the BTF-powered features to be used with kernel modules: > tp_btf, fentry/fexit/fmod_ret, lsm, bpf_iter, etc. I love to see this work going forward. My/Our (+Saeed +Ahern) use-case is for NIC-driver kernel modules. I want drivers to define a BTF struct that describe a meta-data area that can be consumed/used by XDP, also available during xdp_frame to SKB transition, which happens in net-core. So, I hope BTF-IDs are also "available" from core kernel code? > This patch set is a pre-requisite to adding split BTF support to pahole, which > is a prerequisite to integrating split BTF into the Linux kernel build setup > to generate BTF for kernel modules. The latter will come as a follow-up patch > series once this series makes it to the libbpf and pahole makes use of it. > > Patch #4 introduces necessary basic support for split BTF into libbpf APIs. > Patch #8 implements minimal changes to BTF dedup algorithm to allow > deduplicating split BTFs. Patch #11 adds extra -B flag to bpftool to allow to > specify the path to base BTF for cases when one wants to dump or inspect split > BTF. All the rest are refactorings, clean ups, bug fixes and selftests. > > v1->v2: > - addressed Song's feedback.
On Thu, Nov 5, 2020 at 1:53 AM Jesper Dangaard Brouer <brouer@redhat.com> wrote: > > On Wed, 4 Nov 2020 20:33:50 -0800 > Andrii Nakryiko <andrii@kernel.org> wrote: > > > This patch set adds support for generating and deduplicating split BTF. This > > is an enhancement to the BTF, which allows to designate one BTF as the "base > > BTF" (e.g., vmlinux BTF), and one or more other BTFs as "split BTF" (e.g., > > kernel module BTF), which are building upon and extending base BTF with extra > > types and strings. > > > > Once loaded, split BTF appears as a single unified BTF superset of base BTF, > > with continuous and transparent numbering scheme. This allows all the existing > > users of BTF to work correctly and stay agnostic to the base/split BTFs > > composition. The only difference is in how to instantiate split BTF: it > > requires base BTF to be alread instantiated and passed to btf__new_xxx_split() > > or btf__parse_xxx_split() "constructors" explicitly. > > > > This split approach is necessary if we are to have a reasonably-sized kernel > > module BTFs. By deduping each kernel module's BTF individually, resulting > > module BTFs contain copies of a lot of kernel types that are already present > > in vmlinux BTF. Even those single copies result in a big BTF size bloat. On my > > kernel configuration with 700 modules built, non-split BTF approach results in > > 115MBs of BTFs across all modules. With split BTF deduplication approach, > > total size is down to 5.2MBs total, which is on part with vmlinux BTF (at > > around 4MBs). This seems reasonable and practical. As to why we'd need kernel > > module BTFs, that should be pretty obvious to anyone using BPF at this point, > > as it allows all the BTF-powered features to be used with kernel modules: > > tp_btf, fentry/fexit/fmod_ret, lsm, bpf_iter, etc. > > I love to see this work going forward. > Thanks. > My/Our (+Saeed +Ahern) use-case is for NIC-driver kernel modules. I > want drivers to define a BTF struct that describe a meta-data area that > can be consumed/used by XDP, also available during xdp_frame to SKB > transition, which happens in net-core. So, I hope BTF-IDs are also > "available" from core kernel code? I'll probably need a more specific example to understand what exactly you are asking and how you see everything working together, sorry. If you are asking about support for using BTF_ID_LIST() macro in a kernel module, then right now we don't call resolve_btfids on modules, so it's not supported there yet. It's trivial to add, but we'll probably need to teach resolve_btfids to understand split BTF. We can do that separately after the basic "infra" lands, though. > > > > This patch set is a pre-requisite to adding split BTF support to pahole, which > > is a prerequisite to integrating split BTF into the Linux kernel build setup > > to generate BTF for kernel modules. The latter will come as a follow-up patch > > series once this series makes it to the libbpf and pahole makes use of it. > > > > Patch #4 introduces necessary basic support for split BTF into libbpf APIs. > > Patch #8 implements minimal changes to BTF dedup algorithm to allow > > deduplicating split BTFs. Patch #11 adds extra -B flag to bpftool to allow to > > specify the path to base BTF for cases when one wants to dump or inspect split > > BTF. All the rest are refactorings, clean ups, bug fixes and selftests. > > > > v1->v2: > > - addressed Song's feedback. > -- > Best regards, > Jesper Dangaard Brouer > MSc.CS, Principal Kernel Engineer at Red Hat > LinkedIn: http://www.linkedin.com/in/brouer >
On Thu, 2020-11-05 at 11:16 -0800, Andrii Nakryiko wrote: > > > This split approach is necessary if we are to have a reasonably- > > > sized kernel > > > module BTFs. By deduping each kernel module's BTF individually, > > > resulting > > > module BTFs contain copies of a lot of kernel types that are > > > already present > > > in vmlinux BTF. Even those single copies result in a big BTF size > > > bloat. On my > > > kernel configuration with 700 modules built, non-split BTF > > > approach results in > > > 115MBs of BTFs across all modules. With split BTF deduplication > > > approach, > > > total size is down to 5.2MBs total, which is on part with vmlinux > > > BTF (at > > > around 4MBs). This seems reasonable and practical. As to why we'd > > > need kernel > > > module BTFs, that should be pretty obvious to anyone using BPF at > > > this point, > > > as it allows all the BTF-powered features to be used with kernel > > > modules: > > > tp_btf, fentry/fexit/fmod_ret, lsm, bpf_iter, etc. > > I love to see this work going forward. > > > Thanks. > > > > > My/Our (+Saeed +Ahern) use-case is for NIC-driver kernel modules. > > I > > want drivers to define a BTF struct that describe a meta-data area > > that > > can be consumed/used by XDP, also available during xdp_frame to SKB > > transition, which happens in net-core. So, I hope BTF-IDs are also > > "available" from core kernel code? > > > I'll probably need a more specific example to understand what exactly > > you are asking and how you see everything working together, sorry. > > BTF-IDs can be made available for kernel/drivers, I've wrote a small patch for this a while ago. https://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux.git/commit/?h=topic/xdp_metadata3&id=6c1cb83629226889d6fadd3ba694e827fca3e247 So the basic use case is that : 1- driver kernel/registers a BTF format (one or more). 2- Userland queries driver's registered BTF to be able understand the kernel/driver buffers format. driver example of using this infrastructure: https://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux.git/commit/?h=topic/xdp_metadata3&id=9c24657d6cb3a7852c2e948dc9782f3f39b60104 User Queries driver's XDP metadata BTF format: https://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux.git/commit/?h=topic/xdp_metadata3&id=6a117e2d9196f58de7cf067741e84ec242af27f6 Dump it as C header style https://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux.git/commit/?h=topic/xdp_metadata3&id= 8bd99626879bff28379707ac3a2c3bb94fd5b410 And then use it in your XDP program to parse packets meta data passed from this specific driver. ( i mean no real parsing is required, you just point to the meta data buffer with the metadata btf formatted C strucuter). > > If you are asking about support for using BTF_ID_LIST() macro in a > > kernel module, then right now we don't call resolve_btfids on > modules, > > so it's not supported there yet. It's trivial to add, but we'll > > probably need to teach resolve_btfids to understand split BTF. We can > > do that separately after the basic "infra" lands, though.
On Thu, Nov 5, 2020 at 11:38 AM Saeed Mahameed <saeedm@nvidia.com> wrote: > > On Thu, 2020-11-05 at 11:16 -0800, Andrii Nakryiko wrote: > > > > This split approach is necessary if we are to have a reasonably- > > > > sized kernel > > > > module BTFs. By deduping each kernel module's BTF individually, > > > > resulting > > > > module BTFs contain copies of a lot of kernel types that are > > > > already present > > > > in vmlinux BTF. Even those single copies result in a big BTF size > > > > bloat. On my > > > > kernel configuration with 700 modules built, non-split BTF > > > > approach results in > > > > 115MBs of BTFs across all modules. With split BTF deduplication > > > > approach, > > > > total size is down to 5.2MBs total, which is on part with vmlinux > > > > BTF (at > > > > around 4MBs). This seems reasonable and practical. As to why we'd > > > > need kernel > > > > module BTFs, that should be pretty obvious to anyone using BPF at > > > > this point, > > > > as it allows all the BTF-powered features to be used with kernel > > > > modules: > > > > tp_btf, fentry/fexit/fmod_ret, lsm, bpf_iter, etc. > > > I love to see this work going forward. > > > > > > Thanks. > > > > > > > > > My/Our (+Saeed +Ahern) use-case is for NIC-driver kernel modules. > > > I > > > want drivers to define a BTF struct that describe a meta-data area > > > that > > > can be consumed/used by XDP, also available during xdp_frame to SKB > > > transition, which happens in net-core. So, I hope BTF-IDs are also > > > "available" from core kernel code? > > > > > > I'll probably need a more specific example to understand what exactly > > > > you are asking and how you see everything working together, sorry. > > > > > > BTF-IDs can be made available for kernel/drivers, I've wrote a small > patch for this a while ago. > > https://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux.git/commit/?h=topic/xdp_metadata3&id=6c1cb83629226889d6fadd3ba694e827fca3e247 > > So the basic use case is that : > 1- driver kernel/registers a BTF format (one or more). This is now not needed, it happens automatically for module BTF. > 2- Userland queries driver's registered BTF to be able understand the > kernel/driver buffers format. Here the module might need to know its BTF's ID, in addition to BTF type ID. Or maybe it doesn't. User-space tools can just access BTF from /sys/kernel/btf/module_name and use provided BTF type ID to dump whatever is necessary. > > driver example of using this infrastructure: > https://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux.git/commit/?h=topic/xdp_metadata3&id=9c24657d6cb3a7852c2e948dc9782f3f39b60104 This, thankfully, won't be needed, you'll just have a normal C struct and it will be just present in module's BTF. > > User Queries driver's XDP metadata BTF format: > https://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux.git/commit/?h=topic/xdp_metadata3&id=6a117e2d9196f58de7cf067741e84ec242af27f6 For this we need support for BTF_ID macro for modules. As I said, it's pretty easy to add, but feel free to contribute this once the basic infra lands. > > Dump it as C header style > https://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux.git/commit/?h=topic/xdp_metadata3&id= > 8bd99626879bff28379707ac3a2c3bb94fd5b410 This is available as libbpf-provided API now (see btf_dump APIs). And bpftool has support to dump all BTF types as C definitions as well. You might want to do something a bit more targeted, but that's details. > > And then use it in your XDP program to parse packets meta data passed > from this specific driver. ( i mean no real parsing is required, you > just point to the meta data buffer with the metadata btf formatted C > strucuter). > > > > > > If you are asking about support for using BTF_ID_LIST() macro in a > > > > kernel module, then right now we don't call resolve_btfids on > > modules, > > > > so it's not supported there yet. It's trivial to add, but we'll > > > > probably need to teach resolve_btfids to understand split BTF. We can > > > > do that separately after the basic "infra" lands, though. > >
Hello: This series was applied to bpf/bpf-next.git (refs/heads/master): On Wed, 4 Nov 2020 20:33:50 -0800 you wrote: > This patch set adds support for generating and deduplicating split BTF. This > is an enhancement to the BTF, which allows to designate one BTF as the "base > BTF" (e.g., vmlinux BTF), and one or more other BTFs as "split BTF" (e.g., > kernel module BTF), which are building upon and extending base BTF with extra > types and strings. > > Once loaded, split BTF appears as a single unified BTF superset of base BTF, > with continuous and transparent numbering scheme. This allows all the existing > users of BTF to work correctly and stay agnostic to the base/split BTFs > composition. The only difference is in how to instantiate split BTF: it > requires base BTF to be alread instantiated and passed to btf__new_xxx_split() > or btf__parse_xxx_split() "constructors" explicitly. > > [...] Here is the summary with links: - [v2,bpf-next,01/11] libbpf: factor out common operations in BTF writing APIs https://git.kernel.org/bpf/bpf-next/c/c81ed6d81e05 - [v2,bpf-next,02/11] selftest/bpf: relax btf_dedup test checks https://git.kernel.org/bpf/bpf-next/c/d9448f94962b - [v2,bpf-next,03/11] libbpf: unify and speed up BTF string deduplication https://git.kernel.org/bpf/bpf-next/c/88a82c2a9ab5 - [v2,bpf-next,04/11] libbpf: implement basic split BTF support https://git.kernel.org/bpf/bpf-next/c/ba451366bf44 - [v2,bpf-next,05/11] selftests/bpf: add split BTF basic test https://git.kernel.org/bpf/bpf-next/c/197389da2fbf - [v2,bpf-next,06/11] selftests/bpf: add checking of raw type dump in BTF writer APIs selftests https://git.kernel.org/bpf/bpf-next/c/1306c980cf89 - [v2,bpf-next,07/11] libbpf: fix BTF data layout checks and allow empty BTF https://git.kernel.org/bpf/bpf-next/c/d8123624506c - [v2,bpf-next,08/11] libbpf: support BTF dedup of split BTFs https://git.kernel.org/bpf/bpf-next/c/f86524efcf9e - [v2,bpf-next,09/11] libbpf: accomodate DWARF/compiler bug with duplicated identical arrays https://git.kernel.org/bpf/bpf-next/c/6b6e6b1d09aa - [v2,bpf-next,10/11] selftests/bpf: add split BTF dedup selftests https://git.kernel.org/bpf/bpf-next/c/232338fa2fb4 - [v2,bpf-next,11/11] tools/bpftool: add bpftool support for split BTF https://git.kernel.org/bpf/bpf-next/c/75fa1777694c You are awesome, thank you! -- Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html
This patch set adds support for generating and deduplicating split BTF. This is an enhancement to the BTF, which allows to designate one BTF as the "base BTF" (e.g., vmlinux BTF), and one or more other BTFs as "split BTF" (e.g., kernel module BTF), which are building upon and extending base BTF with extra types and strings. Once loaded, split BTF appears as a single unified BTF superset of base BTF, with continuous and transparent numbering scheme. This allows all the existing users of BTF to work correctly and stay agnostic to the base/split BTFs composition. The only difference is in how to instantiate split BTF: it requires base BTF to be alread instantiated and passed to btf__new_xxx_split() or btf__parse_xxx_split() "constructors" explicitly. This split approach is necessary if we are to have a reasonably-sized kernel module BTFs. By deduping each kernel module's BTF individually, resulting module BTFs contain copies of a lot of kernel types that are already present in vmlinux BTF. Even those single copies result in a big BTF size bloat. On my kernel configuration with 700 modules built, non-split BTF approach results in 115MBs of BTFs across all modules. With split BTF deduplication approach, total size is down to 5.2MBs total, which is on part with vmlinux BTF (at around 4MBs). This seems reasonable and practical. As to why we'd need kernel module BTFs, that should be pretty obvious to anyone using BPF at this point, as it allows all the BTF-powered features to be used with kernel modules: tp_btf, fentry/fexit/fmod_ret, lsm, bpf_iter, etc. This patch set is a pre-requisite to adding split BTF support to pahole, which is a prerequisite to integrating split BTF into the Linux kernel build setup to generate BTF for kernel modules. The latter will come as a follow-up patch series once this series makes it to the libbpf and pahole makes use of it. Patch #4 introduces necessary basic support for split BTF into libbpf APIs. Patch #8 implements minimal changes to BTF dedup algorithm to allow deduplicating split BTFs. Patch #11 adds extra -B flag to bpftool to allow to specify the path to base BTF for cases when one wants to dump or inspect split BTF. All the rest are refactorings, clean ups, bug fixes and selftests. v1->v2: - addressed Song's feedback. Andrii Nakryiko (11): libbpf: factor out common operations in BTF writing APIs selftest/bpf: relax btf_dedup test checks libbpf: unify and speed up BTF string deduplication libbpf: implement basic split BTF support selftests/bpf: add split BTF basic test selftests/bpf: add checking of raw type dump in BTF writer APIs selftests libbpf: fix BTF data layout checks and allow empty BTF libbpf: support BTF dedup of split BTFs libbpf: accomodate DWARF/compiler bug with duplicated identical arrays selftests/bpf: add split BTF dedup selftests tools/bpftool: add bpftool support for split BTF tools/bpf/bpftool/btf.c | 9 +- tools/bpf/bpftool/main.c | 15 +- tools/bpf/bpftool/main.h | 1 + tools/lib/bpf/btf.c | 807 ++++++++++-------- tools/lib/bpf/btf.h | 8 + tools/lib/bpf/libbpf.map | 9 + tools/testing/selftests/bpf/Makefile | 2 +- tools/testing/selftests/bpf/btf_helpers.c | 259 ++++++ tools/testing/selftests/bpf/btf_helpers.h | 19 + tools/testing/selftests/bpf/prog_tests/btf.c | 40 +- .../bpf/prog_tests/btf_dedup_split.c | 325 +++++++ .../selftests/bpf/prog_tests/btf_split.c | 99 +++ .../selftests/bpf/prog_tests/btf_write.c | 43 + tools/testing/selftests/bpf/test_progs.h | 11 + 14 files changed, 1292 insertions(+), 355 deletions(-) create mode 100644 tools/testing/selftests/bpf/btf_helpers.c create mode 100644 tools/testing/selftests/bpf/btf_helpers.h create mode 100644 tools/testing/selftests/bpf/prog_tests/btf_dedup_split.c create mode 100644 tools/testing/selftests/bpf/prog_tests/btf_split.c