[bpf-next,00/10] CO-RE offset relocations

Message ID	20190724192742.1419254-1-andriin@fb.com
Headers	show Return-Path: <bpf-owner@vger.kernel.org> Smtp-Origin-Hostprefix: dev From: Andrii Nakryiko <andriin@fb.com> Smtp-Origin-Hostname: dev101.prn2.facebook.com To: <bpf@vger.kernel.org>, <netdev@vger.kernel.org>, <ast@fb.com>, <daniel@iogearbox.net>, <yhs@fb.com> CC: <andrii.nakryiko@gmail.com>, <kernel-team@fb.com>, Andrii Nakryiko <andriin@fb.com> Smtp-Origin-Cluster: prn2c23 Subject: [PATCH bpf-next 00/10] CO-RE offset relocations Date: Wed, 24 Jul 2019 12:27:32 -0700 Message-ID: <20190724192742.1419254-1-andriin@fb.com> MIME-Version: 1.0 Content-Type: text/plain Sender: bpf-owner@vger.kernel.org Precedence: bulk
Series	CO-RE offset relocations \| expand [bpf-next,00/10] CO-RE offset relocations [bpf-next,01/10] libbpf: add .BTF.ext offset relocation section loading [bpf-next,02/10] libbpf: implement BPF CO-RE offset relocation algorithm [bpf-next,03/10] selftests/bpf: add CO-RE relocs testing setup [bpf-next,04/10] selftests/bpf: add CO-RE relocs struct flavors tests [bpf-next,05/10] selftests/bpf: add CO-RE relocs nesting tests [bpf-next,06/10] selftests/bpf: add CO-RE relocs array tests [bpf-next,07/10] selftests/bpf: add CO-RE relocs enum/ptr/func_proto tests [bpf-next,08/10] selftests/bpf: add CO-RE relocs modifiers/typedef tests [bpf-next,09/10] selftest/bpf: add CO-RE relocs ptr-as-array tests [bpf-next,10/10] selftests/bpf: add CO-RE relocs ints tests

Message ID

20190724192742.1419254-1-andriin@fb.com

Headers

Smtp-Origin-Hostprefix: dev
From: Andrii Nakryiko <andriin@fb.com>
Smtp-Origin-Hostname: dev101.prn2.facebook.com
To: <bpf@vger.kernel.org>, <netdev@vger.kernel.org>, <ast@fb.com>,
	<daniel@iogearbox.net>, <yhs@fb.com>
CC: <andrii.nakryiko@gmail.com>, <kernel-team@fb.com>,
	Andrii Nakryiko <andriin@fb.com>
Smtp-Origin-Cluster: prn2c23
Subject: [PATCH bpf-next 00/10] CO-RE offset relocations
Date: Wed, 24 Jul 2019 12:27:32 -0700
Message-ID: <20190724192742.1419254-1-andriin@fb.com>
MIME-Version: 1.0
Content-Type: text/plain
Sender: bpf-owner@vger.kernel.org
Precedence: bulk

Series

CO-RE offset relocations | expand

Message

Andrii Nakryiko July 24, 2019, 7:27 p.m. UTC

This patch set implements central part of CO-RE (Compile Once - Run
Everywhere, see [0] and [1] for slides and video): relocating field offsets.
Most of the details are written down as comments to corresponding parts of the
code.

Patch #1 adds loading of .BTF.ext offset relocations section and macros to
work with its contents.
Patch #2 implements CO-RE relocations algorithm in libbpf.
Patches #3-#10 adds selftests validating various parts of relocation handling,
type compatibility, etc.

For all tests to work, you'll need latest Clang/LLVM supporting
__builtin_preserve_access_index intrinsic, used for recording offset
relocations. Kernel on which selftests run should have BTF information built
in (CONFIG_DEBUG_INFO_BTF=y).

  [0] http://vger.kernel.org/bpfconf2019.html#session-2
  [1] http://vger.kernel.org/lpc-bpf2018.html#session-2CO-RE relocations

This patch set implements central part of CO-RE (Compile Once - Run
Everywhere, see [0] and [1] for slides and video): relocating field offsets.
Most of the details are written down as comments to corresponding parts of the
code.

Patch #1 adds loading of .BTF.ext offset relocations section and macros to
work with its contents.
Patch #2 implements CO-RE relocations algorithm in libbpf.
Patches #3-#10 adds selftests validating various parts of relocation handling,
type compatibility, etc.

For all tests to work, you'll need latest Clang/LLVM supporting
__builtin_preserve_access_index intrinsic, used for recording offset
relocations. Kernel on which selftests run should have BTF information built
in (CONFIG_DEBUG_INFO_BTF=y).

  [0] http://vger.kernel.org/bpfconf2019.html#session-2
  [1] http://vger.kernel.org/lpc-bpf2018.html#session-2

Andrii Nakryiko (10):
  libbpf: add .BTF.ext offset relocation section loading
  libbpf: implement BPF CO-RE offset relocation algorithm
  selftests/bpf: add CO-RE relocs testing setup
  selftests/bpf: add CO-RE relocs struct flavors tests
  selftests/bpf: add CO-RE relocs nesting tests
  selftests/bpf: add CO-RE relocs array tests
  selftests/bpf: add CO-RE relocs enum/ptr/func_proto tests
  selftests/bpf: add CO-RE relocs modifiers/typedef tests
  selftest/bpf: add CO-RE relocs ptr-as-array tests
  selftests/bpf: add CO-RE relocs ints tests

 tools/lib/bpf/btf.c                           |  64 +-
 tools/lib/bpf/btf.h                           |   4 +
 tools/lib/bpf/libbpf.c                        | 866 +++++++++++++++++-
 tools/lib/bpf/libbpf.h                        |   1 +
 tools/lib/bpf/libbpf_internal.h               |  91 ++
 .../selftests/bpf/prog_tests/core_reloc.c     | 363 ++++++++
 .../bpf/progs/btf__core_reloc_arrays.c        |   3 +
 .../btf__core_reloc_arrays___diff_arr_dim.c   |   3 +
 ...btf__core_reloc_arrays___diff_arr_val_sz.c |   3 +
 .../btf__core_reloc_arrays___err_non_array.c  |   3 +
 ...btf__core_reloc_arrays___err_too_shallow.c |   3 +
 .../btf__core_reloc_arrays___err_too_small.c  |   3 +
 ..._core_reloc_arrays___err_wrong_val_type1.c |   3 +
 ..._core_reloc_arrays___err_wrong_val_type2.c |   3 +
 .../bpf/progs/btf__core_reloc_flavors.c       |   3 +
 .../btf__core_reloc_flavors__err_wrong_name.c |   3 +
 .../bpf/progs/btf__core_reloc_ints.c          |   3 +
 .../bpf/progs/btf__core_reloc_ints___bool.c   |   3 +
 .../btf__core_reloc_ints___err_bitfield.c     |   3 +
 .../btf__core_reloc_ints___err_wrong_sz_16.c  |   3 +
 .../btf__core_reloc_ints___err_wrong_sz_32.c  |   3 +
 .../btf__core_reloc_ints___err_wrong_sz_64.c  |   3 +
 .../btf__core_reloc_ints___err_wrong_sz_8.c   |   3 +
 .../btf__core_reloc_ints___reverse_sign.c     |   3 +
 .../bpf/progs/btf__core_reloc_mods.c          |   3 +
 .../progs/btf__core_reloc_mods___mod_swap.c   |   3 +
 .../progs/btf__core_reloc_mods___typedefs.c   |   3 +
 .../bpf/progs/btf__core_reloc_nesting.c       |   3 +
 .../btf__core_reloc_nesting___anon_embed.c    |   3 +
 ...f__core_reloc_nesting___dup_compat_types.c |   5 +
 ...core_reloc_nesting___err_array_container.c |   3 +
 ...tf__core_reloc_nesting___err_array_field.c |   3 +
 ...e_reloc_nesting___err_dup_incompat_types.c |   4 +
 ...re_reloc_nesting___err_missing_container.c |   3 +
 ...__core_reloc_nesting___err_missing_field.c |   3 +
 ..._reloc_nesting___err_nonstruct_container.c |   3 +
 ...e_reloc_nesting___err_partial_match_dups.c |   4 +
 .../btf__core_reloc_nesting___err_too_deep.c  |   3 +
 .../btf__core_reloc_nesting___extra_nesting.c |   3 +
 ..._core_reloc_nesting___struct_union_mixup.c |   3 +
 .../bpf/progs/btf__core_reloc_primitives.c    |   3 +
 ...f__core_reloc_primitives___diff_enum_def.c |   3 +
 ..._core_reloc_primitives___diff_func_proto.c |   3 +
 ...f__core_reloc_primitives___diff_ptr_type.c |   3 +
 ...tf__core_reloc_primitives___err_non_enum.c |   3 +
 ...btf__core_reloc_primitives___err_non_int.c |   3 +
 ...btf__core_reloc_primitives___err_non_ptr.c |   3 +
 .../bpf/progs/btf__core_reloc_ptr_as_arr.c    |   3 +
 .../btf__core_reloc_ptr_as_arr___diff_sz.c    |   3 +
 .../selftests/bpf/progs/core_reloc_types.h    | 642 +++++++++++++
 .../bpf/progs/test_core_reloc_arrays.c        |  58 ++
 .../bpf/progs/test_core_reloc_flavors.c       |  65 ++
 .../bpf/progs/test_core_reloc_ints.c          |  48 +
 .../bpf/progs/test_core_reloc_kernel.c        |  39 +
 .../bpf/progs/test_core_reloc_mods.c          |  68 ++
 .../bpf/progs/test_core_reloc_nesting.c       |  48 +
 .../bpf/progs/test_core_reloc_primitives.c    |  50 +
 .../bpf/progs/test_core_reloc_ptr_as_arr.c    |  34 +
 58 files changed, 2527 insertions(+), 47 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/core_reloc.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___diff_arr_dim.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___diff_arr_val_sz.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___err_non_array.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___err_too_shallow.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___err_too_small.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___err_wrong_val_type1.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___err_wrong_val_type2.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_flavors.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_flavors__err_wrong_name.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___bool.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___err_bitfield.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___err_wrong_sz_16.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___err_wrong_sz_32.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___err_wrong_sz_64.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___err_wrong_sz_8.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___reverse_sign.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_mods.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_mods___mod_swap.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_mods___typedefs.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___anon_embed.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___dup_compat_types.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_array_container.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_array_field.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_dup_incompat_types.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_missing_container.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_missing_field.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_nonstruct_container.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_partial_match_dups.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_too_deep.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___extra_nesting.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___struct_union_mixup.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___diff_enum_def.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___diff_func_proto.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___diff_ptr_type.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___err_non_enum.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___err_non_int.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___err_non_ptr.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ptr_as_arr.c
 create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ptr_as_arr___diff_sz.c
 create mode 100644 tools/testing/selftests/bpf/progs/core_reloc_types.h
 create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_arrays.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_flavors.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_ints.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_kernel.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_mods.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_nesting.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_primitives.c
 create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_ptr_as_arr.c

Comments

Song Liu July 29, 2019, 8:20 p.m. UTC | #1

On Wed, Jul 24, 2019 at 1:34 PM Andrii Nakryiko <andriin@fb.com> wrote:
>
> This patch set implements central part of CO-RE (Compile Once - Run
> Everywhere, see [0] and [1] for slides and video): relocating field offsets.
> Most of the details are written down as comments to corresponding parts of the
> code.
>
> Patch #1 adds loading of .BTF.ext offset relocations section and macros to
> work with its contents.
> Patch #2 implements CO-RE relocations algorithm in libbpf.
> Patches #3-#10 adds selftests validating various parts of relocation handling,
> type compatibility, etc.
>
> For all tests to work, you'll need latest Clang/LLVM supporting
> __builtin_preserve_access_index intrinsic, used for recording offset
> relocations. Kernel on which selftests run should have BTF information built
> in (CONFIG_DEBUG_INFO_BTF=y).
>
>   [0] http://vger.kernel.org/bpfconf2019.html#session-2
>   [1] http://vger.kernel.org/lpc-bpf2018.html#session-2CO-RE relocations
>
> This patch set implements central part of CO-RE (Compile Once - Run
> Everywhere, see [0] and [1] for slides and video): relocating field offsets.
> Most of the details are written down as comments to corresponding parts of the
> code.
>
> Patch #1 adds loading of .BTF.ext offset relocations section and macros to
> work with its contents.
> Patch #2 implements CO-RE relocations algorithm in libbpf.
> Patches #3-#10 adds selftests validating various parts of relocation handling,
> type compatibility, etc.
>
> For all tests to work, you'll need latest Clang/LLVM supporting
> __builtin_preserve_access_index intrinsic, used for recording offset
> relocations. Kernel on which selftests run should have BTF information built
> in (CONFIG_DEBUG_INFO_BTF=y).
>
>   [0] http://vger.kernel.org/bpfconf2019.html#session-2
>   [1] http://vger.kernel.org/lpc-bpf2018.html#session-2
>
> Andrii Nakryiko (10):
>   libbpf: add .BTF.ext offset relocation section loading
>   libbpf: implement BPF CO-RE offset relocation algorithm
>   selftests/bpf: add CO-RE relocs testing setup
>   selftests/bpf: add CO-RE relocs struct flavors tests
>   selftests/bpf: add CO-RE relocs nesting tests
>   selftests/bpf: add CO-RE relocs array tests
>   selftests/bpf: add CO-RE relocs enum/ptr/func_proto tests
>   selftests/bpf: add CO-RE relocs modifiers/typedef tests
>   selftest/bpf: add CO-RE relocs ptr-as-array tests
>   selftests/bpf: add CO-RE relocs ints tests
>
>  tools/lib/bpf/btf.c                           |  64 +-
>  tools/lib/bpf/btf.h                           |   4 +
>  tools/lib/bpf/libbpf.c                        | 866 +++++++++++++++++-
>  tools/lib/bpf/libbpf.h                        |   1 +
>  tools/lib/bpf/libbpf_internal.h               |  91 ++
>  .../selftests/bpf/prog_tests/core_reloc.c     | 363 ++++++++
>  .../bpf/progs/btf__core_reloc_arrays.c        |   3 +
>  .../btf__core_reloc_arrays___diff_arr_dim.c   |   3 +
>  ...btf__core_reloc_arrays___diff_arr_val_sz.c |   3 +
>  .../btf__core_reloc_arrays___err_non_array.c  |   3 +
>  ...btf__core_reloc_arrays___err_too_shallow.c |   3 +
>  .../btf__core_reloc_arrays___err_too_small.c  |   3 +
>  ..._core_reloc_arrays___err_wrong_val_type1.c |   3 +
>  ..._core_reloc_arrays___err_wrong_val_type2.c |   3 +
>  .../bpf/progs/btf__core_reloc_flavors.c       |   3 +
>  .../btf__core_reloc_flavors__err_wrong_name.c |   3 +
>  .../bpf/progs/btf__core_reloc_ints.c          |   3 +
>  .../bpf/progs/btf__core_reloc_ints___bool.c   |   3 +
>  .../btf__core_reloc_ints___err_bitfield.c     |   3 +
>  .../btf__core_reloc_ints___err_wrong_sz_16.c  |   3 +
>  .../btf__core_reloc_ints___err_wrong_sz_32.c  |   3 +
>  .../btf__core_reloc_ints___err_wrong_sz_64.c  |   3 +
>  .../btf__core_reloc_ints___err_wrong_sz_8.c   |   3 +
>  .../btf__core_reloc_ints___reverse_sign.c     |   3 +
>  .../bpf/progs/btf__core_reloc_mods.c          |   3 +
>  .../progs/btf__core_reloc_mods___mod_swap.c   |   3 +
>  .../progs/btf__core_reloc_mods___typedefs.c   |   3 +
>  .../bpf/progs/btf__core_reloc_nesting.c       |   3 +
>  .../btf__core_reloc_nesting___anon_embed.c    |   3 +
>  ...f__core_reloc_nesting___dup_compat_types.c |   5 +
>  ...core_reloc_nesting___err_array_container.c |   3 +
>  ...tf__core_reloc_nesting___err_array_field.c |   3 +
>  ...e_reloc_nesting___err_dup_incompat_types.c |   4 +
>  ...re_reloc_nesting___err_missing_container.c |   3 +
>  ...__core_reloc_nesting___err_missing_field.c |   3 +
>  ..._reloc_nesting___err_nonstruct_container.c |   3 +
>  ...e_reloc_nesting___err_partial_match_dups.c |   4 +
>  .../btf__core_reloc_nesting___err_too_deep.c  |   3 +
>  .../btf__core_reloc_nesting___extra_nesting.c |   3 +
>  ..._core_reloc_nesting___struct_union_mixup.c |   3 +
>  .../bpf/progs/btf__core_reloc_primitives.c    |   3 +
>  ...f__core_reloc_primitives___diff_enum_def.c |   3 +
>  ..._core_reloc_primitives___diff_func_proto.c |   3 +
>  ...f__core_reloc_primitives___diff_ptr_type.c |   3 +
>  ...tf__core_reloc_primitives___err_non_enum.c |   3 +
>  ...btf__core_reloc_primitives___err_non_int.c |   3 +
>  ...btf__core_reloc_primitives___err_non_ptr.c |   3 +
>  .../bpf/progs/btf__core_reloc_ptr_as_arr.c    |   3 +
>  .../btf__core_reloc_ptr_as_arr___diff_sz.c    |   3 +
>  .../selftests/bpf/progs/core_reloc_types.h    | 642 +++++++++++++
>  .../bpf/progs/test_core_reloc_arrays.c        |  58 ++
>  .../bpf/progs/test_core_reloc_flavors.c       |  65 ++
>  .../bpf/progs/test_core_reloc_ints.c          |  48 +
>  .../bpf/progs/test_core_reloc_kernel.c        |  39 +
>  .../bpf/progs/test_core_reloc_mods.c          |  68 ++
>  .../bpf/progs/test_core_reloc_nesting.c       |  48 +
>  .../bpf/progs/test_core_reloc_primitives.c    |  50 +
>  .../bpf/progs/test_core_reloc_ptr_as_arr.c    |  34 +
>  58 files changed, 2527 insertions(+), 47 deletions(-)
>  create mode 100644 tools/testing/selftests/bpf/prog_tests/core_reloc.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___diff_arr_dim.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___diff_arr_val_sz.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___err_non_array.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___err_too_shallow.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___err_too_small.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___err_wrong_val_type1.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___err_wrong_val_type2.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_flavors.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_flavors__err_wrong_name.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___bool.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___err_bitfield.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___err_wrong_sz_16.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___err_wrong_sz_32.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___err_wrong_sz_64.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___err_wrong_sz_8.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___reverse_sign.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_mods.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_mods___mod_swap.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_mods___typedefs.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___anon_embed.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___dup_compat_types.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_array_container.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_array_field.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_dup_incompat_types.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_missing_container.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_missing_field.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_nonstruct_container.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_partial_match_dups.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_too_deep.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___extra_nesting.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___struct_union_mixup.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___diff_enum_def.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___diff_func_proto.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___diff_ptr_type.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___err_non_enum.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___err_non_int.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___err_non_ptr.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ptr_as_arr.c
>  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ptr_as_arr___diff_sz.c
>  create mode 100644 tools/testing/selftests/bpf/progs/core_reloc_types.h
>  create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_arrays.c
>  create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_flavors.c
>  create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_ints.c
>  create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_kernel.c
>  create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_mods.c
>  create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_nesting.c
>  create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_primitives.c
>  create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_ptr_as_arr.c

We have created a lot of small files. Would it be cleaner if we can
somehow put these
data in one file (maybe different sections?).

Alternatively, maybe create a folder for these files:
  tools/testing/selftests/bpf/progs/core/

Thanks,
Song

Song Liu July 29, 2019, 8:36 p.m. UTC | #2

On Mon, Jul 29, 2019 at 1:20 PM Song Liu <liu.song.a23@gmail.com> wrote:
>
> On Wed, Jul 24, 2019 at 1:34 PM Andrii Nakryiko <andriin@fb.com> wrote:
> >
> > This patch set implements central part of CO-RE (Compile Once - Run
> > Everywhere, see [0] and [1] for slides and video): relocating field offsets.
> > Most of the details are written down as comments to corresponding parts of the
> > code.
> >
> > Patch #1 adds loading of .BTF.ext offset relocations section and macros to
> > work with its contents.
> > Patch #2 implements CO-RE relocations algorithm in libbpf.
> > Patches #3-#10 adds selftests validating various parts of relocation handling,
> > type compatibility, etc.
> >
> > For all tests to work, you'll need latest Clang/LLVM supporting
> > __builtin_preserve_access_index intrinsic, used for recording offset
> > relocations. Kernel on which selftests run should have BTF information built
> > in (CONFIG_DEBUG_INFO_BTF=y).
> >
> >   [0] http://vger.kernel.org/bpfconf2019.html#session-2
> >   [1] http://vger.kernel.org/lpc-bpf2018.html#session-2CO-RE relocations
> >
> > This patch set implements central part of CO-RE (Compile Once - Run
> > Everywhere, see [0] and [1] for slides and video): relocating field offsets.
> > Most of the details are written down as comments to corresponding parts of the
> > code.
> >
> > Patch #1 adds loading of .BTF.ext offset relocations section and macros to
> > work with its contents.
> > Patch #2 implements CO-RE relocations algorithm in libbpf.
> > Patches #3-#10 adds selftests validating various parts of relocation handling,
> > type compatibility, etc.
> >
> > For all tests to work, you'll need latest Clang/LLVM supporting
> > __builtin_preserve_access_index intrinsic, used for recording offset
> > relocations. Kernel on which selftests run should have BTF information built
> > in (CONFIG_DEBUG_INFO_BTF=y).
> >
> >   [0] http://vger.kernel.org/bpfconf2019.html#session-2
> >   [1] http://vger.kernel.org/lpc-bpf2018.html#session-2
> >
> > Andrii Nakryiko (10):
> >   libbpf: add .BTF.ext offset relocation section loading
> >   libbpf: implement BPF CO-RE offset relocation algorithm
> >   selftests/bpf: add CO-RE relocs testing setup
> >   selftests/bpf: add CO-RE relocs struct flavors tests
> >   selftests/bpf: add CO-RE relocs nesting tests
> >   selftests/bpf: add CO-RE relocs array tests
> >   selftests/bpf: add CO-RE relocs enum/ptr/func_proto tests
> >   selftests/bpf: add CO-RE relocs modifiers/typedef tests
> >   selftest/bpf: add CO-RE relocs ptr-as-array tests
> >   selftests/bpf: add CO-RE relocs ints tests
> >
> >  tools/lib/bpf/btf.c                           |  64 +-
> >  tools/lib/bpf/btf.h                           |   4 +
> >  tools/lib/bpf/libbpf.c                        | 866 +++++++++++++++++-
> >  tools/lib/bpf/libbpf.h                        |   1 +
> >  tools/lib/bpf/libbpf_internal.h               |  91 ++
> >  .../selftests/bpf/prog_tests/core_reloc.c     | 363 ++++++++
> >  .../bpf/progs/btf__core_reloc_arrays.c        |   3 +
> >  .../btf__core_reloc_arrays___diff_arr_dim.c   |   3 +
> >  ...btf__core_reloc_arrays___diff_arr_val_sz.c |   3 +
> >  .../btf__core_reloc_arrays___err_non_array.c  |   3 +
> >  ...btf__core_reloc_arrays___err_too_shallow.c |   3 +
> >  .../btf__core_reloc_arrays___err_too_small.c  |   3 +
> >  ..._core_reloc_arrays___err_wrong_val_type1.c |   3 +
> >  ..._core_reloc_arrays___err_wrong_val_type2.c |   3 +
> >  .../bpf/progs/btf__core_reloc_flavors.c       |   3 +
> >  .../btf__core_reloc_flavors__err_wrong_name.c |   3 +
> >  .../bpf/progs/btf__core_reloc_ints.c          |   3 +
> >  .../bpf/progs/btf__core_reloc_ints___bool.c   |   3 +
> >  .../btf__core_reloc_ints___err_bitfield.c     |   3 +
> >  .../btf__core_reloc_ints___err_wrong_sz_16.c  |   3 +
> >  .../btf__core_reloc_ints___err_wrong_sz_32.c  |   3 +
> >  .../btf__core_reloc_ints___err_wrong_sz_64.c  |   3 +
> >  .../btf__core_reloc_ints___err_wrong_sz_8.c   |   3 +
> >  .../btf__core_reloc_ints___reverse_sign.c     |   3 +
> >  .../bpf/progs/btf__core_reloc_mods.c          |   3 +
> >  .../progs/btf__core_reloc_mods___mod_swap.c   |   3 +
> >  .../progs/btf__core_reloc_mods___typedefs.c   |   3 +
> >  .../bpf/progs/btf__core_reloc_nesting.c       |   3 +
> >  .../btf__core_reloc_nesting___anon_embed.c    |   3 +
> >  ...f__core_reloc_nesting___dup_compat_types.c |   5 +
> >  ...core_reloc_nesting___err_array_container.c |   3 +
> >  ...tf__core_reloc_nesting___err_array_field.c |   3 +
> >  ...e_reloc_nesting___err_dup_incompat_types.c |   4 +
> >  ...re_reloc_nesting___err_missing_container.c |   3 +
> >  ...__core_reloc_nesting___err_missing_field.c |   3 +
> >  ..._reloc_nesting___err_nonstruct_container.c |   3 +
> >  ...e_reloc_nesting___err_partial_match_dups.c |   4 +
> >  .../btf__core_reloc_nesting___err_too_deep.c  |   3 +
> >  .../btf__core_reloc_nesting___extra_nesting.c |   3 +
> >  ..._core_reloc_nesting___struct_union_mixup.c |   3 +
> >  .../bpf/progs/btf__core_reloc_primitives.c    |   3 +
> >  ...f__core_reloc_primitives___diff_enum_def.c |   3 +
> >  ..._core_reloc_primitives___diff_func_proto.c |   3 +
> >  ...f__core_reloc_primitives___diff_ptr_type.c |   3 +
> >  ...tf__core_reloc_primitives___err_non_enum.c |   3 +
> >  ...btf__core_reloc_primitives___err_non_int.c |   3 +
> >  ...btf__core_reloc_primitives___err_non_ptr.c |   3 +
> >  .../bpf/progs/btf__core_reloc_ptr_as_arr.c    |   3 +
> >  .../btf__core_reloc_ptr_as_arr___diff_sz.c    |   3 +
> >  .../selftests/bpf/progs/core_reloc_types.h    | 642 +++++++++++++
> >  .../bpf/progs/test_core_reloc_arrays.c        |  58 ++
> >  .../bpf/progs/test_core_reloc_flavors.c       |  65 ++
> >  .../bpf/progs/test_core_reloc_ints.c          |  48 +
> >  .../bpf/progs/test_core_reloc_kernel.c        |  39 +
> >  .../bpf/progs/test_core_reloc_mods.c          |  68 ++
> >  .../bpf/progs/test_core_reloc_nesting.c       |  48 +
> >  .../bpf/progs/test_core_reloc_primitives.c    |  50 +
> >  .../bpf/progs/test_core_reloc_ptr_as_arr.c    |  34 +
> >  58 files changed, 2527 insertions(+), 47 deletions(-)
> >  create mode 100644 tools/testing/selftests/bpf/prog_tests/core_reloc.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___diff_arr_dim.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___diff_arr_val_sz.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___err_non_array.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___err_too_shallow.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___err_too_small.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___err_wrong_val_type1.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___err_wrong_val_type2.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_flavors.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_flavors__err_wrong_name.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___bool.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___err_bitfield.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___err_wrong_sz_16.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___err_wrong_sz_32.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___err_wrong_sz_64.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___err_wrong_sz_8.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___reverse_sign.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_mods.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_mods___mod_swap.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_mods___typedefs.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___anon_embed.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___dup_compat_types.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_array_container.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_array_field.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_dup_incompat_types.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_missing_container.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_missing_field.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_nonstruct_container.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_partial_match_dups.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_too_deep.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___extra_nesting.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___struct_union_mixup.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___diff_enum_def.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___diff_func_proto.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___diff_ptr_type.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___err_non_enum.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___err_non_int.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___err_non_ptr.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ptr_as_arr.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ptr_as_arr___diff_sz.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/core_reloc_types.h
> >  create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_arrays.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_flavors.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_ints.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_kernel.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_mods.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_nesting.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_primitives.c
> >  create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_ptr_as_arr.c
>
> We have created a lot of small files. Would it be cleaner if we can
> somehow put these
> data in one file (maybe different sections?).

After reading more, I guess you have tried this and end up with current
design: keep most struct defines in core_reloc_types.h.

>
> Alternatively, maybe create a folder for these files:
>   tools/testing/selftests/bpf/progs/core/

I guess this would still make it cleaner.

Thanks,
Song

Andrii Nakryiko July 29, 2019, 11:09 p.m. UTC | #3

On Mon, Jul 29, 2019 at 1:37 PM Song Liu <liu.song.a23@gmail.com> wrote:
>
> On Mon, Jul 29, 2019 at 1:20 PM Song Liu <liu.song.a23@gmail.com> wrote:
> >
> > On Wed, Jul 24, 2019 at 1:34 PM Andrii Nakryiko <andriin@fb.com> wrote:
> > >
> > > This patch set implements central part of CO-RE (Compile Once - Run
> > > Everywhere, see [0] and [1] for slides and video): relocating field offsets.
> > > Most of the details are written down as comments to corresponding parts of the
> > > code.
> > >
> > > Patch #1 adds loading of .BTF.ext offset relocations section and macros to
> > > work with its contents.
> > > Patch #2 implements CO-RE relocations algorithm in libbpf.
> > > Patches #3-#10 adds selftests validating various parts of relocation handling,
> > > type compatibility, etc.
> > >
> > > For all tests to work, you'll need latest Clang/LLVM supporting
> > > __builtin_preserve_access_index intrinsic, used for recording offset
> > > relocations. Kernel on which selftests run should have BTF information built
> > > in (CONFIG_DEBUG_INFO_BTF=y).
> > >
> > >   [0] http://vger.kernel.org/bpfconf2019.html#session-2
> > >   [1] http://vger.kernel.org/lpc-bpf2018.html#session-2CO-RE relocations
> > >
> > > This patch set implements central part of CO-RE (Compile Once - Run
> > > Everywhere, see [0] and [1] for slides and video): relocating field offsets.
> > > Most of the details are written down as comments to corresponding parts of the
> > > code.
> > >
> > > Patch #1 adds loading of .BTF.ext offset relocations section and macros to
> > > work with its contents.
> > > Patch #2 implements CO-RE relocations algorithm in libbpf.
> > > Patches #3-#10 adds selftests validating various parts of relocation handling,
> > > type compatibility, etc.
> > >
> > > For all tests to work, you'll need latest Clang/LLVM supporting
> > > __builtin_preserve_access_index intrinsic, used for recording offset
> > > relocations. Kernel on which selftests run should have BTF information built
> > > in (CONFIG_DEBUG_INFO_BTF=y).
> > >
> > >   [0] http://vger.kernel.org/bpfconf2019.html#session-2
> > >   [1] http://vger.kernel.org/lpc-bpf2018.html#session-2
> > >
> > > Andrii Nakryiko (10):
> > >   libbpf: add .BTF.ext offset relocation section loading
> > >   libbpf: implement BPF CO-RE offset relocation algorithm
> > >   selftests/bpf: add CO-RE relocs testing setup
> > >   selftests/bpf: add CO-RE relocs struct flavors tests
> > >   selftests/bpf: add CO-RE relocs nesting tests
> > >   selftests/bpf: add CO-RE relocs array tests
> > >   selftests/bpf: add CO-RE relocs enum/ptr/func_proto tests
> > >   selftests/bpf: add CO-RE relocs modifiers/typedef tests
> > >   selftest/bpf: add CO-RE relocs ptr-as-array tests
> > >   selftests/bpf: add CO-RE relocs ints tests
> > >
> > >  tools/lib/bpf/btf.c                           |  64 +-
> > >  tools/lib/bpf/btf.h                           |   4 +
> > >  tools/lib/bpf/libbpf.c                        | 866 +++++++++++++++++-
> > >  tools/lib/bpf/libbpf.h                        |   1 +
> > >  tools/lib/bpf/libbpf_internal.h               |  91 ++
> > >  .../selftests/bpf/prog_tests/core_reloc.c     | 363 ++++++++
> > >  .../bpf/progs/btf__core_reloc_arrays.c        |   3 +
> > >  .../btf__core_reloc_arrays___diff_arr_dim.c   |   3 +
> > >  ...btf__core_reloc_arrays___diff_arr_val_sz.c |   3 +
> > >  .../btf__core_reloc_arrays___err_non_array.c  |   3 +
> > >  ...btf__core_reloc_arrays___err_too_shallow.c |   3 +
> > >  .../btf__core_reloc_arrays___err_too_small.c  |   3 +
> > >  ..._core_reloc_arrays___err_wrong_val_type1.c |   3 +
> > >  ..._core_reloc_arrays___err_wrong_val_type2.c |   3 +
> > >  .../bpf/progs/btf__core_reloc_flavors.c       |   3 +
> > >  .../btf__core_reloc_flavors__err_wrong_name.c |   3 +
> > >  .../bpf/progs/btf__core_reloc_ints.c          |   3 +
> > >  .../bpf/progs/btf__core_reloc_ints___bool.c   |   3 +
> > >  .../btf__core_reloc_ints___err_bitfield.c     |   3 +
> > >  .../btf__core_reloc_ints___err_wrong_sz_16.c  |   3 +
> > >  .../btf__core_reloc_ints___err_wrong_sz_32.c  |   3 +
> > >  .../btf__core_reloc_ints___err_wrong_sz_64.c  |   3 +
> > >  .../btf__core_reloc_ints___err_wrong_sz_8.c   |   3 +
> > >  .../btf__core_reloc_ints___reverse_sign.c     |   3 +
> > >  .../bpf/progs/btf__core_reloc_mods.c          |   3 +
> > >  .../progs/btf__core_reloc_mods___mod_swap.c   |   3 +
> > >  .../progs/btf__core_reloc_mods___typedefs.c   |   3 +
> > >  .../bpf/progs/btf__core_reloc_nesting.c       |   3 +
> > >  .../btf__core_reloc_nesting___anon_embed.c    |   3 +
> > >  ...f__core_reloc_nesting___dup_compat_types.c |   5 +
> > >  ...core_reloc_nesting___err_array_container.c |   3 +
> > >  ...tf__core_reloc_nesting___err_array_field.c |   3 +
> > >  ...e_reloc_nesting___err_dup_incompat_types.c |   4 +
> > >  ...re_reloc_nesting___err_missing_container.c |   3 +
> > >  ...__core_reloc_nesting___err_missing_field.c |   3 +
> > >  ..._reloc_nesting___err_nonstruct_container.c |   3 +
> > >  ...e_reloc_nesting___err_partial_match_dups.c |   4 +
> > >  .../btf__core_reloc_nesting___err_too_deep.c  |   3 +
> > >  .../btf__core_reloc_nesting___extra_nesting.c |   3 +
> > >  ..._core_reloc_nesting___struct_union_mixup.c |   3 +
> > >  .../bpf/progs/btf__core_reloc_primitives.c    |   3 +
> > >  ...f__core_reloc_primitives___diff_enum_def.c |   3 +
> > >  ..._core_reloc_primitives___diff_func_proto.c |   3 +
> > >  ...f__core_reloc_primitives___diff_ptr_type.c |   3 +
> > >  ...tf__core_reloc_primitives___err_non_enum.c |   3 +
> > >  ...btf__core_reloc_primitives___err_non_int.c |   3 +
> > >  ...btf__core_reloc_primitives___err_non_ptr.c |   3 +
> > >  .../bpf/progs/btf__core_reloc_ptr_as_arr.c    |   3 +
> > >  .../btf__core_reloc_ptr_as_arr___diff_sz.c    |   3 +
> > >  .../selftests/bpf/progs/core_reloc_types.h    | 642 +++++++++++++
> > >  .../bpf/progs/test_core_reloc_arrays.c        |  58 ++
> > >  .../bpf/progs/test_core_reloc_flavors.c       |  65 ++
> > >  .../bpf/progs/test_core_reloc_ints.c          |  48 +
> > >  .../bpf/progs/test_core_reloc_kernel.c        |  39 +
> > >  .../bpf/progs/test_core_reloc_mods.c          |  68 ++
> > >  .../bpf/progs/test_core_reloc_nesting.c       |  48 +
> > >  .../bpf/progs/test_core_reloc_primitives.c    |  50 +
> > >  .../bpf/progs/test_core_reloc_ptr_as_arr.c    |  34 +
> > >  58 files changed, 2527 insertions(+), 47 deletions(-)
> > >  create mode 100644 tools/testing/selftests/bpf/prog_tests/core_reloc.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___diff_arr_dim.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___diff_arr_val_sz.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___err_non_array.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___err_too_shallow.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___err_too_small.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___err_wrong_val_type1.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___err_wrong_val_type2.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_flavors.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_flavors__err_wrong_name.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___bool.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___err_bitfield.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___err_wrong_sz_16.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___err_wrong_sz_32.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___err_wrong_sz_64.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___err_wrong_sz_8.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___reverse_sign.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_mods.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_mods___mod_swap.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_mods___typedefs.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___anon_embed.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___dup_compat_types.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_array_container.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_array_field.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_dup_incompat_types.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_missing_container.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_missing_field.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_nonstruct_container.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_partial_match_dups.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_too_deep.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___extra_nesting.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___struct_union_mixup.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___diff_enum_def.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___diff_func_proto.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___diff_ptr_type.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___err_non_enum.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___err_non_int.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___err_non_ptr.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ptr_as_arr.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ptr_as_arr___diff_sz.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/core_reloc_types.h
> > >  create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_arrays.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_flavors.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_ints.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_kernel.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_mods.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_nesting.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_primitives.c
> > >  create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_ptr_as_arr.c
> >
> > We have created a lot of small files. Would it be cleaner if we can
> > somehow put these
> > data in one file (maybe different sections?).
>
> After reading more, I guess you have tried this and end up with current
> design: keep most struct defines in core_reloc_types.h.

Yeah, I have all the definition in one header file, but then I need
individual combinations as separate BTFs, so I essentially "pick"
desired types using function declarations. Creating those BTFs by hand
would be a nightmare to create and maintain.

>
> >
> > Alternatively, maybe create a folder for these files:
> >   tools/testing/selftests/bpf/progs/core/
>
> I guess this would still make it cleaner.

There is nothing too special about core tests to split them. Also it
would require Makefile changes and would deviate test_progs
definitions from analogous test_maps, test_verifier, test_btf, etc, so
I'm not sure about that. I though about putting those btf__* files
under separate directory, but I'm on the fence there as well, as I'd
rather have related files to stay together...


>
> Thanks,
> Song

Song Liu July 30, 2019, 5:27 a.m. UTC | #4

> On Jul 29, 2019, at 4:09 PM, Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:
> 
> On Mon, Jul 29, 2019 at 1:37 PM Song Liu <liu.song.a23@gmail.com> wrote:
>> 
>> On Mon, Jul 29, 2019 at 1:20 PM Song Liu <liu.song.a23@gmail.com> wrote:
>>> 
>>> On Wed, Jul 24, 2019 at 1:34 PM Andrii Nakryiko <andriin@fb.com> wrote:
>>>> 
>>>> This patch set implements central part of CO-RE (Compile Once - Run
>>>> Everywhere, see [0] and [1] for slides and video): relocating field offsets.
>>>> Most of the details are written down as comments to corresponding parts of the
>>>> code.
>>>> 
>>>> Patch #1 adds loading of .BTF.ext offset relocations section and macros to
>>>> work with its contents.
>>>> Patch #2 implements CO-RE relocations algorithm in libbpf.
>>>> Patches #3-#10 adds selftests validating various parts of relocation handling,
>>>> type compatibility, etc.
>>>> 
>>>> For all tests to work, you'll need latest Clang/LLVM supporting
>>>> __builtin_preserve_access_index intrinsic, used for recording offset
>>>> relocations. Kernel on which selftests run should have BTF information built
>>>> in (CONFIG_DEBUG_INFO_BTF=y).
>>>> 
>>>>  [0] http://vger.kernel.org/bpfconf2019.html#session-2
>>>>  [1] http://vger.kernel.org/lpc-bpf2018.html#session-2CO-RE relocations
>>>> 
>>>> This patch set implements central part of CO-RE (Compile Once - Run
>>>> Everywhere, see [0] and [1] for slides and video): relocating field offsets.
>>>> Most of the details are written down as comments to corresponding parts of the
>>>> code.
>>>> 
>>>> Patch #1 adds loading of .BTF.ext offset relocations section and macros to
>>>> work with its contents.
>>>> Patch #2 implements CO-RE relocations algorithm in libbpf.
>>>> Patches #3-#10 adds selftests validating various parts of relocation handling,
>>>> type compatibility, etc.
>>>> 
>>>> For all tests to work, you'll need latest Clang/LLVM supporting
>>>> __builtin_preserve_access_index intrinsic, used for recording offset
>>>> relocations. Kernel on which selftests run should have BTF information built
>>>> in (CONFIG_DEBUG_INFO_BTF=y).
>>>> 
>>>>  [0] http://vger.kernel.org/bpfconf2019.html#session-2
>>>>  [1] http://vger.kernel.org/lpc-bpf2018.html#session-2
>>>> 
>>>> Andrii Nakryiko (10):
>>>>  libbpf: add .BTF.ext offset relocation section loading
>>>>  libbpf: implement BPF CO-RE offset relocation algorithm
>>>>  selftests/bpf: add CO-RE relocs testing setup
>>>>  selftests/bpf: add CO-RE relocs struct flavors tests
>>>>  selftests/bpf: add CO-RE relocs nesting tests
>>>>  selftests/bpf: add CO-RE relocs array tests
>>>>  selftests/bpf: add CO-RE relocs enum/ptr/func_proto tests
>>>>  selftests/bpf: add CO-RE relocs modifiers/typedef tests
>>>>  selftest/bpf: add CO-RE relocs ptr-as-array tests
>>>>  selftests/bpf: add CO-RE relocs ints tests
>>>> 
>>>> tools/lib/bpf/btf.c                           |  64 +-
>>>> tools/lib/bpf/btf.h                           |   4 +
>>>> tools/lib/bpf/libbpf.c                        | 866 +++++++++++++++++-
>>>> tools/lib/bpf/libbpf.h                        |   1 +
>>>> tools/lib/bpf/libbpf_internal.h               |  91 ++
>>>> .../selftests/bpf/prog_tests/core_reloc.c     | 363 ++++++++
>>>> .../bpf/progs/btf__core_reloc_arrays.c        |   3 +
>>>> .../btf__core_reloc_arrays___diff_arr_dim.c   |   3 +
>>>> ...btf__core_reloc_arrays___diff_arr_val_sz.c |   3 +
>>>> .../btf__core_reloc_arrays___err_non_array.c  |   3 +
>>>> ...btf__core_reloc_arrays___err_too_shallow.c |   3 +
>>>> .../btf__core_reloc_arrays___err_too_small.c  |   3 +
>>>> ..._core_reloc_arrays___err_wrong_val_type1.c |   3 +
>>>> ..._core_reloc_arrays___err_wrong_val_type2.c |   3 +
>>>> .../bpf/progs/btf__core_reloc_flavors.c       |   3 +
>>>> .../btf__core_reloc_flavors__err_wrong_name.c |   3 +
>>>> .../bpf/progs/btf__core_reloc_ints.c          |   3 +
>>>> .../bpf/progs/btf__core_reloc_ints___bool.c   |   3 +
>>>> .../btf__core_reloc_ints___err_bitfield.c     |   3 +
>>>> .../btf__core_reloc_ints___err_wrong_sz_16.c  |   3 +
>>>> .../btf__core_reloc_ints___err_wrong_sz_32.c  |   3 +
>>>> .../btf__core_reloc_ints___err_wrong_sz_64.c  |   3 +
>>>> .../btf__core_reloc_ints___err_wrong_sz_8.c   |   3 +
>>>> .../btf__core_reloc_ints___reverse_sign.c     |   3 +
>>>> .../bpf/progs/btf__core_reloc_mods.c          |   3 +
>>>> .../progs/btf__core_reloc_mods___mod_swap.c   |   3 +
>>>> .../progs/btf__core_reloc_mods___typedefs.c   |   3 +
>>>> .../bpf/progs/btf__core_reloc_nesting.c       |   3 +
>>>> .../btf__core_reloc_nesting___anon_embed.c    |   3 +
>>>> ...f__core_reloc_nesting___dup_compat_types.c |   5 +
>>>> ...core_reloc_nesting___err_array_container.c |   3 +
>>>> ...tf__core_reloc_nesting___err_array_field.c |   3 +
>>>> ...e_reloc_nesting___err_dup_incompat_types.c |   4 +
>>>> ...re_reloc_nesting___err_missing_container.c |   3 +
>>>> ...__core_reloc_nesting___err_missing_field.c |   3 +
>>>> ..._reloc_nesting___err_nonstruct_container.c |   3 +
>>>> ...e_reloc_nesting___err_partial_match_dups.c |   4 +
>>>> .../btf__core_reloc_nesting___err_too_deep.c  |   3 +
>>>> .../btf__core_reloc_nesting___extra_nesting.c |   3 +
>>>> ..._core_reloc_nesting___struct_union_mixup.c |   3 +
>>>> .../bpf/progs/btf__core_reloc_primitives.c    |   3 +
>>>> ...f__core_reloc_primitives___diff_enum_def.c |   3 +
>>>> ..._core_reloc_primitives___diff_func_proto.c |   3 +
>>>> ...f__core_reloc_primitives___diff_ptr_type.c |   3 +
>>>> ...tf__core_reloc_primitives___err_non_enum.c |   3 +
>>>> ...btf__core_reloc_primitives___err_non_int.c |   3 +
>>>> ...btf__core_reloc_primitives___err_non_ptr.c |   3 +
>>>> .../bpf/progs/btf__core_reloc_ptr_as_arr.c    |   3 +
>>>> .../btf__core_reloc_ptr_as_arr___diff_sz.c    |   3 +
>>>> .../selftests/bpf/progs/core_reloc_types.h    | 642 +++++++++++++
>>>> .../bpf/progs/test_core_reloc_arrays.c        |  58 ++
>>>> .../bpf/progs/test_core_reloc_flavors.c       |  65 ++
>>>> .../bpf/progs/test_core_reloc_ints.c          |  48 +
>>>> .../bpf/progs/test_core_reloc_kernel.c        |  39 +
>>>> .../bpf/progs/test_core_reloc_mods.c          |  68 ++
>>>> .../bpf/progs/test_core_reloc_nesting.c       |  48 +
>>>> .../bpf/progs/test_core_reloc_primitives.c    |  50 +
>>>> .../bpf/progs/test_core_reloc_ptr_as_arr.c    |  34 +
>>>> 58 files changed, 2527 insertions(+), 47 deletions(-)
>>>> create mode 100644 tools/testing/selftests/bpf/prog_tests/core_reloc.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___diff_arr_dim.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___diff_arr_val_sz.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___err_non_array.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___err_too_shallow.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___err_too_small.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___err_wrong_val_type1.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_arrays___err_wrong_val_type2.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_flavors.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_flavors__err_wrong_name.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___bool.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___err_bitfield.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___err_wrong_sz_16.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___err_wrong_sz_32.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___err_wrong_sz_64.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___err_wrong_sz_8.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ints___reverse_sign.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_mods.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_mods___mod_swap.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_mods___typedefs.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___anon_embed.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___dup_compat_types.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_array_container.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_array_field.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_dup_incompat_types.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_missing_container.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_missing_field.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_nonstruct_container.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_partial_match_dups.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___err_too_deep.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___extra_nesting.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_nesting___struct_union_mixup.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___diff_enum_def.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___diff_func_proto.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___diff_ptr_type.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___err_non_enum.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___err_non_int.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_primitives___err_non_ptr.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ptr_as_arr.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/btf__core_reloc_ptr_as_arr___diff_sz.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/core_reloc_types.h
>>>> create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_arrays.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_flavors.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_ints.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_kernel.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_mods.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_nesting.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_primitives.c
>>>> create mode 100644 tools/testing/selftests/bpf/progs/test_core_reloc_ptr_as_arr.c
>>> 
>>> We have created a lot of small files. Would it be cleaner if we can
>>> somehow put these
>>> data in one file (maybe different sections?).
>> 
>> After reading more, I guess you have tried this and end up with current
>> design: keep most struct defines in core_reloc_types.h.
> 
> Yeah, I have all the definition in one header file, but then I need
> individual combinations as separate BTFs, so I essentially "pick"
> desired types using function declarations. Creating those BTFs by hand
> would be a nightmare to create and maintain.
> 
>> 
>>> 
>>> Alternatively, maybe create a folder for these files:
>>>  tools/testing/selftests/bpf/progs/core/
>> 
>> I guess this would still make it cleaner.
> 
> There is nothing too special about core tests to split them. Also it
> would require Makefile changes and would deviate test_progs
> definitions from analogous test_maps, test_verifier, test_btf, etc, so
> I'm not sure about that. I though about putting those btf__* files
> under separate directory, but I'm on the fence there as well, as I'd
> rather have related files to stay together...
> 

I guess we can defer this decision to later. So I am OK with current
approach for now. 

Thanks,
Song