[bpf-next,00/13] bpf: introduce function calls

First of all huge thank you to Daniel, John, Jakub, Edward and others who
reviewed multiple iterations of this patch set over the last many months
and to Dave and others who gave critical feedback during netconf/netdev.

The patch is solid enough and we thought through numerous corner cases,
but it's not the end. More followups with code reorg and features to follow.

TLDR: Allow arbitrary function calls from bpf function to another bpf function.

Since the beginning of bpf all bpf programs were represented as a single function
and program authors were forced to use always_inline for all functions
in their C code. That was causing llvm to unnecessary inflate the code size
and forcing developers to move code to header files with little code reuse.

With a bit of additional complexity teach verifier to recognize
arbitrary function calls from one bpf function to another as long as
all of functions are presented to the verifier as a single bpf program.
Extended program layout:
..
r1 = ..    // arg1
r2 = ..    // arg2
call pc+1  // function call pc-relative
exit
.. = r1    // access arg1
.. = r2    // access arg2
..
call pc+20 // second level of function call
...

It allows for better optimized code and finally allows to introduce
the core bpf libraries that can be reused in different projects,
since programs are no longer limited by single elf file.
With function calls bpf can be compiled into multiple .o files.

This patch is the first step. It detects programs that contain
multiple functions and checks that calls between them are valid.
It splits the sequence of bpf instructions (one program) into a set
of bpf functions that call each other. Calls to only known
functions are allowed. Since all functions are presented to
the verifier at once conceptually it is 'static linking'.

Future plans:
- introduce BPF_PROG_TYPE_LIBRARY and allow a set of bpf functions
  to be loaded into the kernel that can be later linked to other
  programs with concrete program types. Aka 'dynamic linking'.

- introduce function pointer type and indirect calls to allow
  bpf functions call other dynamically loaded bpf functions while
  the caller bpf function is already executing. Aka 'runtime linking'.
  This will be more generic and more flexible alternative
  to bpf_tail_calls.

FAQ:
Q: Interpreter and JIT changes mean that new instruction is introduced ?
A: No. The call instruction technically stays the same. Now it can call
   both kernel helpers and other bpf functions.
   Calling convention stays the same as well.
   From uapi point of view the call insn got new 'relocation' BPF_PSEUDO_CALL
   similar to BPF_PSEUDO_MAP_FD 'relocation' of bpf_ldimm64 insn.

Q: What had to change on LLVM side?
A: Trivial LLVM patch to allow calls was applied to upcoming 6.0 release:
   https://reviews.llvm.org/rL318614
   with few bugfixes as well.
   Make sure to build the latest llvm to have bpf_call support.

More details in the patches.

Alexei Starovoitov (12):
  bpf: introduce function calls (function boundaries)
  bpf: introduce function calls (verification)
  selftests/bpf: add verifier tests for bpf_call
  bpf: teach verifier to recognize zero initialized stack
  selftests/bpf: add tests for stack_zero tracking
  libbpf: add support for bpf_call
  selftests/bpf: add bpf_call test
  selftests/bpf: add xdp noinline test
  bpf: add support for bpf_call to interpreter
  bpf: fix net.core.bpf_jit_enable race
  bpf: x64: add JIT support for multi-function programs
  bpf: arm64: add JIT support for multi-function programs

Daniel Borkmann (1):
  selftests/bpf: additional bpf_call tests

 arch/arm/net/bpf_jit_32.c                        |    2 +-
 arch/arm64/net/bpf_jit_comp.c                    |   70 +-
 arch/mips/net/ebpf_jit.c                         |    2 +-
 arch/powerpc/net/bpf_jit_comp64.c                |    2 +-
 arch/s390/net/bpf_jit_comp.c                     |    2 +-
 arch/sparc/net/bpf_jit_comp_64.c                 |    2 +-
 arch/x86/net/bpf_jit_comp.c                      |   49 +-
 include/linux/bpf.h                              |    4 +
 include/linux/bpf_verifier.h                     |   45 +-
 include/linux/filter.h                           |   13 +-
 include/uapi/linux/bpf.h                         |    6 +
 kernel/bpf/core.c                                |  104 +-
 kernel/bpf/disasm.c                              |    8 +-
 kernel/bpf/syscall.c                             |    3 +-
 kernel/bpf/verifier.c                            | 1120 ++++++++++++---
 tools/include/uapi/linux/bpf.h                   |    6 +
 tools/lib/bpf/bpf.h                              |    2 +-
 tools/lib/bpf/libbpf.c                           |  170 ++-
 tools/testing/selftests/bpf/Makefile             |   12 +-
 tools/testing/selftests/bpf/test_l4lb_noinline.c |  473 +++++++
 tools/testing/selftests/bpf/test_progs.c         |   95 +-
 tools/testing/selftests/bpf/test_verifier.c      | 1624 +++++++++++++++++++++-
 tools/testing/selftests/bpf/test_xdp_noinline.c  |  833 +++++++++++
 23 files changed, 4378 insertions(+), 269 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/test_l4lb_noinline.c
 create mode 100644 tools/testing/selftests/bpf/test_xdp_noinline.c

On 12/15/2017 02:55 AM, Alexei Starovoitov wrote:
> First of all huge thank you to Daniel, John, Jakub, Edward and others who
> reviewed multiple iterations of this patch set over the last many months
> and to Dave and others who gave critical feedback during netconf/netdev.
> 
> The patch is solid enough and we thought through numerous corner cases,
> but it's not the end. More followups with code reorg and features to follow.
> 
> TLDR: Allow arbitrary function calls from bpf function to another bpf function.
> 
> Since the beginning of bpf all bpf programs were represented as a single function
> and program authors were forced to use always_inline for all functions
> in their C code. That was causing llvm to unnecessary inflate the code size
> and forcing developers to move code to header files with little code reuse.
> 
> With a bit of additional complexity teach verifier to recognize
> arbitrary function calls from one bpf function to another as long as
> all of functions are presented to the verifier as a single bpf program.
> Extended program layout:
> ..
> r1 = ..    // arg1
> r2 = ..    // arg2
> call pc+1  // function call pc-relative
> exit
> .. = r1    // access arg1
> .. = r2    // access arg2
> ..
> call pc+20 // second level of function call
> ...
> 
> It allows for better optimized code and finally allows to introduce
> the core bpf libraries that can be reused in different projects,
> since programs are no longer limited by single elf file.
> With function calls bpf can be compiled into multiple .o files.
> 
> This patch is the first step. It detects programs that contain
> multiple functions and checks that calls between them are valid.
> It splits the sequence of bpf instructions (one program) into a set
> of bpf functions that call each other. Calls to only known
> functions are allowed. Since all functions are presented to
> the verifier at once conceptually it is 'static linking'.
> 
> Future plans:
> - introduce BPF_PROG_TYPE_LIBRARY and allow a set of bpf functions
>   to be loaded into the kernel that can be later linked to other
>   programs with concrete program types. Aka 'dynamic linking'.
> 
> - introduce function pointer type and indirect calls to allow
>   bpf functions call other dynamically loaded bpf functions while
>   the caller bpf function is already executing. Aka 'runtime linking'.
>   This will be more generic and more flexible alternative
>   to bpf_tail_calls.
> 
> FAQ:
> Q: Interpreter and JIT changes mean that new instruction is introduced ?
> A: No. The call instruction technically stays the same. Now it can call
>    both kernel helpers and other bpf functions.
>    Calling convention stays the same as well.
>    From uapi point of view the call insn got new 'relocation' BPF_PSEUDO_CALL
>    similar to BPF_PSEUDO_MAP_FD 'relocation' of bpf_ldimm64 insn.
> 
> Q: What had to change on LLVM side?
> A: Trivial LLVM patch to allow calls was applied to upcoming 6.0 release:
>    https://reviews.llvm.org/rL318614
>    with few bugfixes as well.
>    Make sure to build the latest llvm to have bpf_call support.
> 
> More details in the patches.

Series applied to bpf-next, thanks Alexei!