| Message ID | 20220512074546.231616-4-hbathini@linux.ibm.com (mailing list archive) |
|---|---|
| State | Changes Requested |
| Headers | show |
| Series | Atomics support for eBPF on powerpc | expand |
On Thu, 2022-05-12 at 13:15 +0530, Hari Bathini wrote: > This adds two atomic opcodes BPF_XCHG and BPF_CMPXCHG on ppc64, both > of which include the BPF_FETCH flag. The kernel's atomic_cmpxchg > operation fundamentally has 3 operands, but we only have two register > fields. Therefore the operand we compare against (the kernel's API > calls it 'old') is hard-coded to be BPF_REG_R0. Also, kernel's > atomic_cmpxchg returns the previous value at dst_reg + off. JIT the > same for BPF too with return value put in BPF_REG_0. > > BPF_REG_R0 = atomic_cmpxchg(dst_reg + off, BPF_REG_R0, src_reg); > > Signed-off-by: Hari Bathini <hbathini@linux.ibm.com> > --- > arch/powerpc/net/bpf_jit_comp64.c | 28 ++++++++++++++++++++++++---- > 1 file changed, 24 insertions(+), 4 deletions(-) > > diff --git a/arch/powerpc/net/bpf_jit_comp64.c > b/arch/powerpc/net/bpf_jit_comp64.c > index 504fa459f9f3..df9e20b22ccb 100644 > --- a/arch/powerpc/net/bpf_jit_comp64.c > +++ b/arch/powerpc/net/bpf_jit_comp64.c > @@ -783,6 +783,9 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 > *image, struct codegen_context * > */ > case BPF_STX | BPF_ATOMIC | BPF_W: > case BPF_STX | BPF_ATOMIC | BPF_DW: > + u32 save_reg = tmp2_reg; > + u32 ret_reg = src_reg; Hi Hari, Some compilers[0][1] don't like these late declarations after case labels: arch/powerpc/net/bpf_jit_comp64.c: In function ‘bpf_jit_build_body’: arch/powerpc/net/bpf_jit_comp64.c:781:4: error: a label can only be part of a statement and a declaration is not a statement u32 save_reg = tmp2_reg; ^~~ arch/powerpc/net/bpf_jit_comp64.c:782:4: error: expected expression before ‘u32’ u32 ret_reg = src_reg; ^~~ arch/powerpc/net/bpf_jit_comp64.c:819:5: error: ‘ret_reg’ undeclared (first use in this function); did you mean ‘dst_reg’? ret_reg = bpf_to_ppc(BPF_REG_0); Adding a semicolon fixes the first issue, i.e. case BPF_STX | BPF_ATOMIC | BPF_DW: ; but then it just complains about mixed declarations and code instead. So you should declare save_reg and ret_reg at the beginning of the for loop like the rest of the variables. - Russell [0]: gcc 5.5.0 https://github.com/ruscur/linux-ci/runs/6418546193?check_suite_focus=true#step:4:122 [1]: clang 12.0 https://github.com/ruscur/linux-ci/runs/6418545338?check_suite_focus=true#step:4:117 > + > /* Get offset into TMP_REG_1 */ > EMIT(PPC_RAW_LI(tmp1_reg, off)); > tmp_idx = ctx->idx * 4; > @@ -813,6 +816,24 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 > *image, struct codegen_context * > case BPF_XOR | BPF_FETCH: > EMIT(PPC_RAW_XOR(tmp2_reg, tmp2_reg, > src_reg)); > break; > + case BPF_CMPXCHG: > + /* > + * Return old value in BPF_REG_0 for > BPF_CMPXCHG & > + * in src_reg for other cases. > + */ > + ret_reg = bpf_to_ppc(BPF_REG_0); > + > + /* Compare with old value in BPF_R0 > */ > + if (size == BPF_DW) > + EMIT(PPC_RAW_CMPD(bpf_to_ppc( > BPF_REG_0), tmp2_reg)); > + else > + EMIT(PPC_RAW_CMPW(bpf_to_ppc( > BPF_REG_0), tmp2_reg)); > + /* Don't set if different from old > value */ > + PPC_BCC_SHORT(COND_NE, (ctx->idx + 3) > * 4); > + fallthrough; > + case BPF_XCHG: > + save_reg = src_reg; > + break; > default: > pr_err_ratelimited( > "eBPF filter atomic op code > %02x (@%d) unsupported\n", > @@ -822,15 +843,14 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 > *image, struct codegen_context * > > /* store new value */ > if (size == BPF_DW) > - EMIT(PPC_RAW_STDCX(tmp2_reg, > tmp1_reg, dst_reg)); > + EMIT(PPC_RAW_STDCX(save_reg, > tmp1_reg, dst_reg)); > else > - EMIT(PPC_RAW_STWCX(tmp2_reg, > tmp1_reg, dst_reg)); > + EMIT(PPC_RAW_STWCX(save_reg, > tmp1_reg, dst_reg)); > /* we're done if this succeeded */ > PPC_BCC_SHORT(COND_NE, tmp_idx); > > - /* For the BPF_FETCH variant, get old value > into src_reg */ > if (imm & BPF_FETCH) > - EMIT(PPC_RAW_MR(src_reg, _R0)); > + EMIT(PPC_RAW_MR(ret_reg, _R0)); > break; > > /*
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c index 504fa459f9f3..df9e20b22ccb 100644 --- a/arch/powerpc/net/bpf_jit_comp64.c +++ b/arch/powerpc/net/bpf_jit_comp64.c @@ -783,6 +783,9 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context * */ case BPF_STX | BPF_ATOMIC | BPF_W: case BPF_STX | BPF_ATOMIC | BPF_DW: + u32 save_reg = tmp2_reg; + u32 ret_reg = src_reg; + /* Get offset into TMP_REG_1 */ EMIT(PPC_RAW_LI(tmp1_reg, off)); tmp_idx = ctx->idx * 4; @@ -813,6 +816,24 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context * case BPF_XOR | BPF_FETCH: EMIT(PPC_RAW_XOR(tmp2_reg, tmp2_reg, src_reg)); break; + case BPF_CMPXCHG: + /* + * Return old value in BPF_REG_0 for BPF_CMPXCHG & + * in src_reg for other cases. + */ + ret_reg = bpf_to_ppc(BPF_REG_0); + + /* Compare with old value in BPF_R0 */ + if (size == BPF_DW) + EMIT(PPC_RAW_CMPD(bpf_to_ppc(BPF_REG_0), tmp2_reg)); + else + EMIT(PPC_RAW_CMPW(bpf_to_ppc(BPF_REG_0), tmp2_reg)); + /* Don't set if different from old value */ + PPC_BCC_SHORT(COND_NE, (ctx->idx + 3) * 4); + fallthrough; + case BPF_XCHG: + save_reg = src_reg; + break; default: pr_err_ratelimited( "eBPF filter atomic op code %02x (@%d) unsupported\n", @@ -822,15 +843,14 @@ int bpf_jit_build_body(struct bpf_prog *fp, u32 *image, struct codegen_context * /* store new value */ if (size == BPF_DW) - EMIT(PPC_RAW_STDCX(tmp2_reg, tmp1_reg, dst_reg)); + EMIT(PPC_RAW_STDCX(save_reg, tmp1_reg, dst_reg)); else - EMIT(PPC_RAW_STWCX(tmp2_reg, tmp1_reg, dst_reg)); + EMIT(PPC_RAW_STWCX(save_reg, tmp1_reg, dst_reg)); /* we're done if this succeeded */ PPC_BCC_SHORT(COND_NE, tmp_idx); - /* For the BPF_FETCH variant, get old value into src_reg */ if (imm & BPF_FETCH) - EMIT(PPC_RAW_MR(src_reg, _R0)); + EMIT(PPC_RAW_MR(ret_reg, _R0)); break; /*
This adds two atomic opcodes BPF_XCHG and BPF_CMPXCHG on ppc64, both of which include the BPF_FETCH flag. The kernel's atomic_cmpxchg operation fundamentally has 3 operands, but we only have two register fields. Therefore the operand we compare against (the kernel's API calls it 'old') is hard-coded to be BPF_REG_R0. Also, kernel's atomic_cmpxchg returns the previous value at dst_reg + off. JIT the same for BPF too with return value put in BPF_REG_0. BPF_REG_R0 = atomic_cmpxchg(dst_reg + off, BPF_REG_R0, src_reg); Signed-off-by: Hari Bathini <hbathini@linux.ibm.com> --- arch/powerpc/net/bpf_jit_comp64.c | 28 ++++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-)