| Message ID | 20200818223921.2911963-1-andriin@fb.com |
|---|---|
| Headers | show |
| Series | Add support for type-based and enum value-based CO-RE relocations | expand |
On Tue, Aug 18, 2020 at 03:39:12PM -0700, Andrii Nakryiko wrote: > This patch set adds libbpf support to two new classes of CO-RE relocations: > type-based (TYPE_EXISTS/TYPE_SIZE/TYPE_ID_LOCAL/TYPE_ID_TARGET) and enum > value-vased (ENUMVAL_EXISTS/ENUMVAL_VALUE): > > LLVM patches adding these relocation in Clang: > - __builtin_btf_type_id() ([0], [1], [2]); > - __builtin_preserve_type_info(), __builtin_preserve_enum_value() ([3], [4]). I've applied patches 1-4, since they're somewhat indepedent of new features in 5+. What should be the process to land the rest? Land llvm first and add to bpf/README.rst that certain llvm commmits are necessary to build the tests? But CI will start failing. We can wait for that to be fixed, but I wonder is there way to detect new clang __builtins automatically in selftests and skip them if clang is too old?
On Tue, Aug 18, 2020 at 6:21 PM Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote: > > On Tue, Aug 18, 2020 at 03:39:12PM -0700, Andrii Nakryiko wrote: > > This patch set adds libbpf support to two new classes of CO-RE relocations: > > type-based (TYPE_EXISTS/TYPE_SIZE/TYPE_ID_LOCAL/TYPE_ID_TARGET) and enum > > value-vased (ENUMVAL_EXISTS/ENUMVAL_VALUE): > > > > LLVM patches adding these relocation in Clang: > > - __builtin_btf_type_id() ([0], [1], [2]); > > - __builtin_preserve_type_info(), __builtin_preserve_enum_value() ([3], [4]). > > I've applied patches 1-4, since they're somewhat indepedent of new features in 5+. > What should be the process to land the rest? > Land llvm first and add to bpf/README.rst that certain llvm commmits are necessary > to build the tests? Clang patches landed about two weeks ago, so they are already in Clang nightly builds. libbpf CI should work fine as it uses clang-12 nightly builds. > But CI will start failing. We can wait for that to be fixed, > but I wonder is there way to detect new clang __builtins automatically in > selftests and skip them if clang is too old? There is a way to detect built-ins availability (__has_builtin macro, [0]) from C code. If we want to do it from Makefile, though, we'd need to do feature detection similar to how we did reallocarray and libbpf-elf-mmap detection I just removed in the other patch set :). Then we'll also need to somehow blacklist tests. Maintaining that would be a pain, honestly. So far selftests/bpf assumed the latest Clang, though, so do you think we should change that, or you were worried that patches hadn't landed yet?
On Tue, Aug 18, 2020 at 06:31:51PM -0700, Andrii Nakryiko wrote: > On Tue, Aug 18, 2020 at 6:21 PM Alexei Starovoitov > <alexei.starovoitov@gmail.com> wrote: > > > > On Tue, Aug 18, 2020 at 03:39:12PM -0700, Andrii Nakryiko wrote: > > > This patch set adds libbpf support to two new classes of CO-RE relocations: > > > type-based (TYPE_EXISTS/TYPE_SIZE/TYPE_ID_LOCAL/TYPE_ID_TARGET) and enum > > > value-vased (ENUMVAL_EXISTS/ENUMVAL_VALUE): > > > > > > LLVM patches adding these relocation in Clang: > > > - __builtin_btf_type_id() ([0], [1], [2]); > > > - __builtin_preserve_type_info(), __builtin_preserve_enum_value() ([3], [4]). > > > > I've applied patches 1-4, since they're somewhat indepedent of new features in 5+. > > What should be the process to land the rest? > > Land llvm first and add to bpf/README.rst that certain llvm commmits are necessary > > to build the tests? > > Clang patches landed about two weeks ago, so they are already in Clang > nightly builds. libbpf CI should work fine as it uses clang-12 nightly > builds. > > > > But CI will start failing. We can wait for that to be fixed, > > but I wonder is there way to detect new clang __builtins automatically in > > selftests and skip them if clang is too old? > > There is a way to detect built-ins availability (__has_builtin macro, > [0]) from C code. If we want to do it from Makefile, though, we'd need > to do feature detection similar to how we did reallocarray and > libbpf-elf-mmap detection I just removed in the other patch set :). > Then we'll also need to somehow blacklist tests. Maintaining that > would be a pain, honestly. So far selftests/bpf assumed the latest > Clang, though, so do you think we should change that, or you were > worried that patches hadn't landed yet? I was hoping that libbpf.h can have builtins unconditionally, but selftests can do feature detection automatically and mark them as 'skip'. People have been forever complaining about constant need to upgrade clang. In this case I think the feature is not fundamental enough (unlike the first set of builtins) to force adoption of new clang. If/when we start using these new builtins beyond selftests that would be a different story.
On Tue, Aug 18, 2020 at 6:37 PM Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote: > > On Tue, Aug 18, 2020 at 06:31:51PM -0700, Andrii Nakryiko wrote: > > On Tue, Aug 18, 2020 at 6:21 PM Alexei Starovoitov > > <alexei.starovoitov@gmail.com> wrote: > > > > > > On Tue, Aug 18, 2020 at 03:39:12PM -0700, Andrii Nakryiko wrote: > > > > This patch set adds libbpf support to two new classes of CO-RE relocations: > > > > type-based (TYPE_EXISTS/TYPE_SIZE/TYPE_ID_LOCAL/TYPE_ID_TARGET) and enum > > > > value-vased (ENUMVAL_EXISTS/ENUMVAL_VALUE): > > > > > > > > LLVM patches adding these relocation in Clang: > > > > - __builtin_btf_type_id() ([0], [1], [2]); > > > > - __builtin_preserve_type_info(), __builtin_preserve_enum_value() ([3], [4]). > > > > > > I've applied patches 1-4, since they're somewhat indepedent of new features in 5+. > > > What should be the process to land the rest? > > > Land llvm first and add to bpf/README.rst that certain llvm commmits are necessary > > > to build the tests? > > > > Clang patches landed about two weeks ago, so they are already in Clang > > nightly builds. libbpf CI should work fine as it uses clang-12 nightly > > builds. > > > > > > > But CI will start failing. We can wait for that to be fixed, > > > but I wonder is there way to detect new clang __builtins automatically in > > > selftests and skip them if clang is too old? > > > > There is a way to detect built-ins availability (__has_builtin macro, > > [0]) from C code. If we want to do it from Makefile, though, we'd need > > to do feature detection similar to how we did reallocarray and > > libbpf-elf-mmap detection I just removed in the other patch set :). > > Then we'll also need to somehow blacklist tests. Maintaining that > > would be a pain, honestly. So far selftests/bpf assumed the latest > > Clang, though, so do you think we should change that, or you were > > worried that patches hadn't landed yet? > > I was hoping that libbpf.h can have builtins unconditionally, but selftests can > do feature detection automatically and mark them as 'skip'. > People have been forever complaining about constant need to upgrade clang. > In this case I think the feature is not fundamental enough (unlike the first > set of builtins) to force adoption of new clang. > If/when we start using these new builtins beyond selftests > that would be a different story. Ok, took some tinkering to do this for btf_type_id() tests, but everything works now as you described above. Posted v2.
This patch set adds libbpf support to two new classes of CO-RE relocations: type-based (TYPE_EXISTS/TYPE_SIZE/TYPE_ID_LOCAL/TYPE_ID_TARGET) and enum value-vased (ENUMVAL_EXISTS/ENUMVAL_VALUE): - TYPE_EXISTS allows to detect presence in kernel BTF of a locally-recorded BTF type. Useful for feature detection (new functionality often comes with new internal kernel types), as well as handling type renames and bigger refactorings. - TYPE_SIZE allows to get the real size (in bytes) of a specified kernel type. Useful for dumping internal structure as-is through perfbuf or ringbuf. - TYPE_ID_LOCAL/TYPE_ID_TARGET allow to capture BTF type ID of a BTF type in program's BTF or kernel BTF, respectively. These could be used for high-performance and space-efficient generic data dumping/logging by relying on small and cheap BTF type ID as a data layout descriptor, for post-processing on user-space side. - ENUMVAL_EXISTS can be used for detecting the presence of enumerator value in kernel's enum type. Most direct application is to detect BPF helper support in kernel. - ENUMVAL_VALUE allows to relocate real integer value of kernel enumerator value, which is subject to change (e.g., always a potential issue for internal, non-UAPI, kernel enums). I've indicated potential applications for these relocations, but relocations themselves are generic and unassuming and are designed to work correctly even in unintended applications. Furthermore, relocated values become constants, known to the verifier and could and would be used for dead branch code detection and elimination. This makes them ideal to do all sorts of feature detection and guarding functionality that's not available on some older (but still supported by BPF program) kernels, while having to compile and maintain one unified source code. As part of this patch set, one potential issue with ambiguous CO-RE relocations was solved (see patch #3). There are also some improvements to the way debug relocation logs are emitted, helping to get a high-level idea of what's going on for users that are willing to dive deeper into the internals of libbpf (or libbpf contributors, of course). Selftests are added for all the new features and relocation ambiguity issue is excercised as well. LLVM patches adding these relocation in Clang: - __builtin_btf_type_id() ([0], [1], [2]); - __builtin_preserve_type_info(), __builtin_preserve_enum_value() ([3], [4]). rfc -> v1: - I've rebased patches on top of series [5], there is a very minor conflict in bpf_core_read.h due to comment change. Series [5] would most probably go in first anyway. [0] https://reviews.llvm.org/D74572 [1] https://reviews.llvm.org/D74668 [2] https://reviews.llvm.org/D85174 [3] https://reviews.llvm.org/D83878 [4] https://reviews.llvm.org/D83242 [5] https://patchwork.ozlabs.org/project/netdev/list/?series=196308&state=* Andrii Nakryiko (9): libbpf: improve error logging for mismatched BTF kind cases libbpf: clean up and improve CO-RE reloc logging libbpf: improve relocation ambiguity detection selftests/bpf: add test validating failure on ambiguous relocation value libbpf: implement type-based CO-RE relocations support selftests/bpf: test TYPE_EXISTS and TYPE_SIZE CO-RE relocations selftests/bpf: add CO-RE relo test for TYPE_ID_LOCAL/TYPE_ID_TARGET libbpf: implement enum value-based CO-RE relocations selftests/bpf: add tests for ENUMVAL_EXISTS/ENUMVAL_VALUE relocations tools/lib/bpf/bpf_core_read.h | 80 +- tools/lib/bpf/btf.c | 17 +- tools/lib/bpf/btf.h | 38 - tools/lib/bpf/libbpf.c | 754 ++++++++++++++---- tools/lib/bpf/libbpf_internal.h | 84 +- .../selftests/bpf/prog_tests/core_reloc.c | 328 +++++++- .../bpf/progs/btf__core_reloc_enumval.c | 3 + .../progs/btf__core_reloc_enumval___diff.c | 3 + .../btf__core_reloc_enumval___err_missing.c | 3 + .../btf__core_reloc_enumval___val3_missing.c | 3 + .../btf__core_reloc_size___err_ambiguous.c | 4 + .../bpf/progs/btf__core_reloc_type_based.c | 3 + ...btf__core_reloc_type_based___all_missing.c | 3 + .../btf__core_reloc_type_based___diff_sz.c | 3 + ...f__core_reloc_type_based___fn_wrong_args.c | 3 + .../btf__core_reloc_type_based___incompat.c | 3 + .../bpf/progs/btf__core_reloc_type_id.c | 3 + ...tf__core_reloc_type_id___missing_targets.c | 3 + .../selftests/bpf/progs/core_reloc_types.h | 352 +++++++- .../bpf/progs/test_core_reloc_enumval.c | 69 ++ .../bpf/progs/test_core_reloc_type_based.c | 107 +++ .../bpf/progs/test_core_reloc_type_id.c | 94 +++ 22 files changed, 1727 insertions(+), 233 deletions(-) create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_enumval.c create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_enumval___diff.c create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_enumval___err_missing.c create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_enumval___val3_missing.c create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_size___err_ambiguous.c create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_type_based.c create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_type_based___all_missing.c create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_type_based___diff_sz.c create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_type_based___fn_wrong_args.c create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_type_based___incompat.c create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_type_id.c create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_type_id___missing_targets.c create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_enumval.c create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_type_based.c create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_type_id.c