From patchwork Mon Dec 16 09:13:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 1210162 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="RwRGeVm1"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 47bwYw1RfRz9sQp for ; Mon, 16 Dec 2019 20:13:56 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727056AbfLPJNz (ORCPT ); Mon, 16 Dec 2019 04:13:55 -0500 Received: from mail-pg1-f194.google.com ([209.85.215.194]:34832 "EHLO mail-pg1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726313AbfLPJNz (ORCPT ); Mon, 16 Dec 2019 04:13:55 -0500 Received: by mail-pg1-f194.google.com with SMTP id l24so3330854pgk.2; Mon, 16 Dec 2019 01:13:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=bYazlit0M4yBFQMfwCbOmC5g6AIzkoA/91yiEDZt7b0=; b=RwRGeVm1aeN2qkJLadfLujw+Gg4mmYcLdCef5IRe1XA0xhjwByucNBCckRiVYY6fLO 4296nh+AjY2dK6wlGr86OM9pf7cguH6tlpQf7mVCmE9ZJhFyMvWHH1X5jR8DAxXPdrzK Pq2mcImd24XixdeDeiReoUuU4bH2acto7KmAoYFdMkqUMLAKgT2mJIi3syGlZU4BFC3P TSSxkz5RTMxufDP3glc76F5k57NP/Cxj5MleehiJXmq63EI0qxK7T7GEVKH6gXaZmUFM qbH4Z0myz4VWg3u95JgmzKBXo23KfOmUgkcIZRXNLngGdkYKAf8Vguh9We/lv8iqrVqW D2lA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=bYazlit0M4yBFQMfwCbOmC5g6AIzkoA/91yiEDZt7b0=; b=D3hX6XSBV6FFVSuKxCr31mXnql7hupfJEr1dhdRv0ZDJwJk3edjnSe8ZLcYqeSIP8P 1hluffqGxQUXjP6vI4c6BoX9EKH650/4t7+OzB6cAvswsy47Atb1lkAD5wSTfJxYM6Si LM+7nojKxd2/peySVJjlDdsPfJlZd3Aiw1GhzV6LDqNSWhz+QlS8pSZtAIHYm9apHAiF rNq38M8idp4IE8nJRCv/9R49MTdcTHsq+hRig7NgV9V6WDQLjpMVR0JqPJgZpVuMbs+w 2Vf/BZMmb1+anZAWd9BQYGTYQKu3NxxRlqSbL09pClyg/EBuEh4jmrxKGM3welIfzv9w oH7A== X-Gm-Message-State: APjAAAVF56i/0EI6WKZHMxD6LTaIazhZXlPBDusFlvb5FuPi+4YwKNkV E8D1I4yni4vMde3utFhn2NULE8JGZWg= X-Google-Smtp-Source: APXvYqya/ZK/sigSlCk224VfrvfZoRMNqXimX/XMn+BCFUGM3rt7mHAstLVa1xEVZ3DJ9X5w/O85hA== X-Received: by 2002:a63:5807:: with SMTP id m7mr15787129pgb.83.1576487634904; Mon, 16 Dec 2019 01:13:54 -0800 (PST) Received: from btopel-mobl.ger.intel.com (fmdmzpr04-ext.fm.intel.com. [192.55.55.39]) by smtp.gmail.com with ESMTPSA id x21sm12505033pfn.164.2019.12.16.01.13.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Dec 2019 01:13:54 -0800 (PST) From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: daniel@iogearbox.net, ast@kernel.org, netdev@vger.kernel.org Cc: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , linux-riscv@lists.infradead.org, bpf@vger.kernel.org Subject: [PATCH bpf-next v2 1/9] riscv, bpf: fix broken BPF tail calls Date: Mon, 16 Dec 2019 10:13:35 +0100 Message-Id: <20191216091343.23260-2-bjorn.topel@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191216091343.23260-1-bjorn.topel@gmail.com> References: <20191216091343.23260-1-bjorn.topel@gmail.com> MIME-Version: 1.0 Sender: bpf-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org The BPF JIT incorrectly clobbered the a0 register, and did not flag usage of s5 register when BPF stack was being used. Fixes: 2353ecc6f91f ("bpf, riscv: add BPF JIT for RV64G") Signed-off-by: Björn Töpel --- arch/riscv/net/bpf_jit_comp.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/arch/riscv/net/bpf_jit_comp.c b/arch/riscv/net/bpf_jit_comp.c index 5451ef3845f2..1606ebd49666 100644 --- a/arch/riscv/net/bpf_jit_comp.c +++ b/arch/riscv/net/bpf_jit_comp.c @@ -120,6 +120,11 @@ static bool seen_reg(int reg, struct rv_jit_context *ctx) return false; } +static void mark_fp(struct rv_jit_context *ctx) +{ + __set_bit(RV_CTX_F_SEEN_S5, &ctx->flags); +} + static void mark_call(struct rv_jit_context *ctx) { __set_bit(RV_CTX_F_SEEN_CALL, &ctx->flags); @@ -596,7 +601,8 @@ static void __build_epilogue(u8 reg, struct rv_jit_context *ctx) emit(rv_addi(RV_REG_SP, RV_REG_SP, stack_adjust), ctx); /* Set return value. */ - emit(rv_addi(RV_REG_A0, RV_REG_A5, 0), ctx); + if (reg == RV_REG_RA) + emit(rv_addi(RV_REG_A0, RV_REG_A5, 0), ctx); emit(rv_jalr(RV_REG_ZERO, reg, 0), ctx); } @@ -1426,6 +1432,10 @@ static void build_prologue(struct rv_jit_context *ctx) { int stack_adjust = 0, store_offset, bpf_stack_adjust; + bpf_stack_adjust = round_up(ctx->prog->aux->stack_depth, 16); + if (bpf_stack_adjust) + mark_fp(ctx); + if (seen_reg(RV_REG_RA, ctx)) stack_adjust += 8; stack_adjust += 8; /* RV_REG_FP */ @@ -1443,7 +1453,6 @@ static void build_prologue(struct rv_jit_context *ctx) stack_adjust += 8; stack_adjust = round_up(stack_adjust, 16); - bpf_stack_adjust = round_up(ctx->prog->aux->stack_depth, 16); stack_adjust += bpf_stack_adjust; store_offset = stack_adjust - 8; From patchwork Mon Dec 16 09:13:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 1210164 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="SGfveANl"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 47bwZ11N46z9sQp for ; Mon, 16 Dec 2019 20:14:01 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727112AbfLPJN7 (ORCPT ); Mon, 16 Dec 2019 04:13:59 -0500 Received: from mail-pf1-f196.google.com ([209.85.210.196]:39341 "EHLO mail-pf1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727086AbfLPJN7 (ORCPT ); Mon, 16 Dec 2019 04:13:59 -0500 Received: by mail-pf1-f196.google.com with SMTP id 2so5242757pfx.6; Mon, 16 Dec 2019 01:13:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=IKFdvlWbIsqVMfKHWS/eTT8rPSF1iVyjiBiv16n+QTk=; b=SGfveANlxHXwLMslfZEaRSW/nPAhYZ6SWMDtMpv0P7BzgNymGsdpDF85JuMVRGqABz E71+C9XlkLs1G9LTkfW8sB7y9MMI/CUVkkyMyUwhY6hTwnCgFWRhmH+dRwOy27geCc1y kTfR591B16StnsAuUsS0G4W6VqD/J2G22gyVJvaDAJhSTMdseHZ/wu2XOHz9EbwDrOax 2X61taMctG07feNkyDP1MWgk0ho1fMG/oisx1d0NqcmBF6EZnWNPCLKbB3/f6krO5SJB ShX6WPG9gRVA/+LgDnjz6z0uUwo7f6ui+Zq9wy3CoU/lk/aODBUEGYMVXdvl/k5tKYt0 wqSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=IKFdvlWbIsqVMfKHWS/eTT8rPSF1iVyjiBiv16n+QTk=; b=ZjN2R60uSJpOzrG106b1zxh7WSTFX0bqFi/9YRG1SZZ9AMEJjvrDEXAEWbP30rns3c GDr2I/5TKA5PtzbYrZZPZy/b3wPJb7Opps7wQegIbWTXgSbRwJ4kPsfRmOMktq7dr/Tc fiW+5UPOKyYoStCSi/vA4e4DIsiNFOOTzPi/eDi3rewbm+lxlP+DtFoYIKhow5Hp91td OqOGBpR3J/hgpUFxWjmTl9xc+TDBogi89HR8T6OghmAYB8BU4pR/nD8wQJMkzYwJTibT UVRHkmqx5GjSGy2Z9nCMx2E/FKfmbuLFMD3tEVYlAjmyF6hg6BTyxXwuTM4/JB/1ERf0 xMjQ== X-Gm-Message-State: APjAAAW8PLxAygLijsnZlNPMjvggtAg4IMWWX8wkS9+7PkfpbFpb4SpI /78RsC4K7yEyni1HwWtk2xs= X-Google-Smtp-Source: APXvYqxfvROp5uFKPkUAnVkaSA6bxX57635PyRj+vzQOEtM3gj4b2rYQUukNxytA1z7gXWYQRgwgJQ== X-Received: by 2002:a63:512:: with SMTP id 18mr16957496pgf.221.1576487638048; Mon, 16 Dec 2019 01:13:58 -0800 (PST) Received: from btopel-mobl.ger.intel.com (fmdmzpr04-ext.fm.intel.com. [192.55.55.39]) by smtp.gmail.com with ESMTPSA id x21sm12505033pfn.164.2019.12.16.01.13.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Dec 2019 01:13:57 -0800 (PST) From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: daniel@iogearbox.net, ast@kernel.org, netdev@vger.kernel.org Cc: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , linux-riscv@lists.infradead.org, bpf@vger.kernel.org, Luke Nelson , Xi Wang Subject: [PATCH bpf-next v2 2/9] riscv, bpf: add support for far branching Date: Mon, 16 Dec 2019 10:13:36 +0100 Message-Id: <20191216091343.23260-3-bjorn.topel@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191216091343.23260-1-bjorn.topel@gmail.com> References: <20191216091343.23260-1-bjorn.topel@gmail.com> MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org This commit adds branch relaxation to the BPF JIT, and with that support for far (offset greater than 12b) branching. The branch relaxation requires more than two passes to converge. For most programs it is three passes, but for larger programs it can be more. Reviewed-by: Luke Nelson Cc: Xi Wang Signed-off-by: Björn Töpel --- arch/riscv/net/bpf_jit_comp.c | 352 ++++++++++++++++++---------------- 1 file changed, 188 insertions(+), 164 deletions(-) diff --git a/arch/riscv/net/bpf_jit_comp.c b/arch/riscv/net/bpf_jit_comp.c index 1606ebd49666..e599458a9bcd 100644 --- a/arch/riscv/net/bpf_jit_comp.c +++ b/arch/riscv/net/bpf_jit_comp.c @@ -461,6 +461,11 @@ static u32 rv_amoadd_d(u8 rd, u8 rs2, u8 rs1, u8 aq, u8 rl) return rv_amo_insn(0, aq, rl, rs2, rs1, 3, rd, 0x2f); } +static u32 rv_auipc(u8 rd, u32 imm31_12) +{ + return rv_u_insn(imm31_12, rd, 0x17); +} + static bool is_12b_int(s64 val) { return -(1 << 11) <= val && val < (1 << 11); @@ -484,7 +489,7 @@ static bool is_32b_int(s64 val) static int is_12b_check(int off, int insn) { if (!is_12b_int(off)) { - pr_err("bpf-jit: insn=%d offset=%d not supported yet!\n", + pr_err("bpf-jit: insn=%d 12b < offset=%d not supported yet!\n", insn, (int)off); return -1; } @@ -494,7 +499,7 @@ static int is_12b_check(int off, int insn) static int is_13b_check(int off, int insn) { if (!is_13b_int(off)) { - pr_err("bpf-jit: insn=%d offset=%d not supported yet!\n", + pr_err("bpf-jit: insn=%d 13b < offset=%d not supported yet!\n", insn, (int)off); return -1; } @@ -504,7 +509,7 @@ static int is_13b_check(int off, int insn) static int is_21b_check(int off, int insn) { if (!is_21b_int(off)) { - pr_err("bpf-jit: insn=%d offset=%d not supported yet!\n", + pr_err("bpf-jit: insn=%d 21b < offset=%d not supported yet!\n", insn, (int)off); return -1; } @@ -550,10 +555,13 @@ static void emit_imm(u8 rd, s64 val, struct rv_jit_context *ctx) emit(rv_addi(rd, rd, lower), ctx); } -static int rv_offset(int bpf_to, int bpf_from, struct rv_jit_context *ctx) +static int rv_offset(int insn, int off, struct rv_jit_context *ctx) { - int from = ctx->offset[bpf_from] - 1, to = ctx->offset[bpf_to]; + int from, to; + off++; /* BPF branch is from PC+1, RV is from PC */ + from = (insn > 0) ? ctx->offset[insn - 1] : 0; + to = (insn + off > 0) ? ctx->offset[insn + off - 1] : 0; return (to - from) << 2; } @@ -606,6 +614,109 @@ static void __build_epilogue(u8 reg, struct rv_jit_context *ctx) emit(rv_jalr(RV_REG_ZERO, reg, 0), ctx); } +/* return -1 or inverted cond */ +static int invert_bpf_cond(u8 cond) +{ + switch (cond) { + case BPF_JEQ: + return BPF_JNE; + case BPF_JGT: + return BPF_JLE; + case BPF_JLT: + return BPF_JGE; + case BPF_JGE: + return BPF_JLT; + case BPF_JLE: + return BPF_JGT; + case BPF_JNE: + return BPF_JEQ; + case BPF_JSGT: + return BPF_JSLE; + case BPF_JSLT: + return BPF_JSGE; + case BPF_JSGE: + return BPF_JSLT; + case BPF_JSLE: + return BPF_JSGT; + } + return -1; +} + +static void emit_bcc(u8 cond, u8 rd, u8 rs, int rvoff, + struct rv_jit_context *ctx) +{ + switch (cond) { + case BPF_JEQ: + emit(rv_beq(rd, rs, rvoff >> 1), ctx); + return; + case BPF_JGT: + emit(rv_bltu(rs, rd, rvoff >> 1), ctx); + return; + case BPF_JLT: + emit(rv_bltu(rd, rs, rvoff >> 1), ctx); + return; + case BPF_JGE: + emit(rv_bgeu(rd, rs, rvoff >> 1), ctx); + return; + case BPF_JLE: + emit(rv_bgeu(rs, rd, rvoff >> 1), ctx); + return; + case BPF_JNE: + emit(rv_bne(rd, rs, rvoff >> 1), ctx); + return; + case BPF_JSGT: + emit(rv_blt(rs, rd, rvoff >> 1), ctx); + return; + case BPF_JSLT: + emit(rv_blt(rd, rs, rvoff >> 1), ctx); + return; + case BPF_JSGE: + emit(rv_bge(rd, rs, rvoff >> 1), ctx); + return; + case BPF_JSLE: + emit(rv_bge(rs, rd, rvoff >> 1), ctx); + } +} + +static void emit_branch(u8 cond, u8 rd, u8 rs, int rvoff, + struct rv_jit_context *ctx) +{ + s64 upper, lower; + + if (is_13b_int(rvoff)) { + emit_bcc(cond, rd, rs, rvoff, ctx); + return; + } + + /* Adjust for jal */ + rvoff -= 4; + + /* Transform, e.g.: + * bne rd,rs,foo + * to + * beq rd,rs,<.L1> + * (auipc foo) + * jal(r) foo + * .L1 + */ + cond = invert_bpf_cond(cond); + if (is_21b_int(rvoff)) { + emit_bcc(cond, rd, rs, 8, ctx); + emit(rv_jal(RV_REG_ZERO, rvoff >> 1), ctx); + return; + } + + /* 32b No need for an additional rvoff adjustment, since we + * get that from the auipc at PC', where PC = PC' + 4. + */ + upper = (rvoff + (1 << 11)) >> 12; + lower = rvoff & 0xfff; + + emit_bcc(cond, rd, rs, 12, ctx); + emit(rv_auipc(RV_REG_T1, upper), ctx); + emit(rv_jalr(RV_REG_ZERO, RV_REG_T1, lower), ctx); +} + static void emit_zext_32(u8 reg, struct rv_jit_context *ctx) { emit(rv_slli(reg, reg, 32), ctx); @@ -693,13 +804,6 @@ static void init_regs(u8 *rd, u8 *rs, const struct bpf_insn *insn, *rs = bpf_to_rv_reg(insn->src_reg, ctx); } -static int rv_offset_check(int *rvoff, s16 off, int insn, - struct rv_jit_context *ctx) -{ - *rvoff = rv_offset(insn + off, insn, ctx); - return is_13b_check(*rvoff, insn); -} - static void emit_zext_32_rd_rs(u8 *rd, u8 *rs, struct rv_jit_context *ctx) { emit(rv_addi(RV_REG_T2, *rd, 0), ctx); @@ -732,13 +836,19 @@ static void emit_sext_32_rd(u8 *rd, struct rv_jit_context *ctx) *rd = RV_REG_T2; } +static bool is_signed_bpf_cond(u8 cond) +{ + return cond == BPF_JSGT || cond == BPF_JSLT || + cond == BPF_JSGE || cond == BPF_JSLE; +} + static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, bool extra_pass) { bool is64 = BPF_CLASS(insn->code) == BPF_ALU64 || BPF_CLASS(insn->code) == BPF_JMP; + int s, e, rvoff, i = insn - ctx->prog->insnsi; struct bpf_prog_aux *aux = ctx->prog->aux; - int rvoff, i = insn - ctx->prog->insnsi; u8 rd = -1, rs = -1, code = insn->code; s16 off = insn->off; s32 imm = insn->imm; @@ -1006,7 +1116,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, /* JUMP off */ case BPF_JMP | BPF_JA: - rvoff = rv_offset(i + off, i, ctx); + rvoff = rv_offset(i, off, ctx); if (!is_21b_int(rvoff)) { pr_err("bpf-jit: insn=%d offset=%d not supported yet!\n", i, rvoff); @@ -1019,194 +1129,96 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, /* IF (dst COND src) JUMP off */ case BPF_JMP | BPF_JEQ | BPF_X: case BPF_JMP32 | BPF_JEQ | BPF_X: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - if (!is64) - emit_zext_32_rd_rs(&rd, &rs, ctx); - emit(rv_beq(rd, rs, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JGT | BPF_X: case BPF_JMP32 | BPF_JGT | BPF_X: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - if (!is64) - emit_zext_32_rd_rs(&rd, &rs, ctx); - emit(rv_bltu(rs, rd, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JLT | BPF_X: case BPF_JMP32 | BPF_JLT | BPF_X: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - if (!is64) - emit_zext_32_rd_rs(&rd, &rs, ctx); - emit(rv_bltu(rd, rs, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JGE | BPF_X: case BPF_JMP32 | BPF_JGE | BPF_X: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - if (!is64) - emit_zext_32_rd_rs(&rd, &rs, ctx); - emit(rv_bgeu(rd, rs, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JLE | BPF_X: case BPF_JMP32 | BPF_JLE | BPF_X: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - if (!is64) - emit_zext_32_rd_rs(&rd, &rs, ctx); - emit(rv_bgeu(rs, rd, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JNE | BPF_X: case BPF_JMP32 | BPF_JNE | BPF_X: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - if (!is64) - emit_zext_32_rd_rs(&rd, &rs, ctx); - emit(rv_bne(rd, rs, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JSGT | BPF_X: case BPF_JMP32 | BPF_JSGT | BPF_X: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - if (!is64) - emit_sext_32_rd_rs(&rd, &rs, ctx); - emit(rv_blt(rs, rd, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JSLT | BPF_X: case BPF_JMP32 | BPF_JSLT | BPF_X: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - if (!is64) - emit_sext_32_rd_rs(&rd, &rs, ctx); - emit(rv_blt(rd, rs, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JSGE | BPF_X: case BPF_JMP32 | BPF_JSGE | BPF_X: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - if (!is64) - emit_sext_32_rd_rs(&rd, &rs, ctx); - emit(rv_bge(rd, rs, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JSLE | BPF_X: case BPF_JMP32 | BPF_JSLE | BPF_X: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - if (!is64) - emit_sext_32_rd_rs(&rd, &rs, ctx); - emit(rv_bge(rs, rd, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JSET | BPF_X: case BPF_JMP32 | BPF_JSET | BPF_X: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - if (!is64) - emit_zext_32_rd_rs(&rd, &rs, ctx); - emit(rv_and(RV_REG_T1, rd, rs), ctx); - emit(rv_bne(RV_REG_T1, RV_REG_ZERO, rvoff >> 1), ctx); + rvoff = rv_offset(i, off, ctx); + if (!is64) { + s = ctx->ninsns; + if (is_signed_bpf_cond(BPF_OP(code))) + emit_sext_32_rd_rs(&rd, &rs, ctx); + else + emit_zext_32_rd_rs(&rd, &rs, ctx); + e = ctx->ninsns; + + /* Adjust for extra insns */ + rvoff -= (e - s) << 2; + } + + if (BPF_OP(code) == BPF_JSET) { + /* Adjust for and */ + rvoff -= 4; + emit(rv_and(RV_REG_T1, rd, rs), ctx); + emit_branch(BPF_JNE, RV_REG_T1, RV_REG_ZERO, rvoff, + ctx); + } else { + emit_branch(BPF_OP(code), rd, rs, rvoff, ctx); + } break; /* IF (dst COND imm) JUMP off */ case BPF_JMP | BPF_JEQ | BPF_K: case BPF_JMP32 | BPF_JEQ | BPF_K: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - emit_imm(RV_REG_T1, imm, ctx); - if (!is64) - emit_zext_32_rd_t1(&rd, ctx); - emit(rv_beq(rd, RV_REG_T1, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JGT | BPF_K: case BPF_JMP32 | BPF_JGT | BPF_K: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - emit_imm(RV_REG_T1, imm, ctx); - if (!is64) - emit_zext_32_rd_t1(&rd, ctx); - emit(rv_bltu(RV_REG_T1, rd, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JLT | BPF_K: case BPF_JMP32 | BPF_JLT | BPF_K: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - emit_imm(RV_REG_T1, imm, ctx); - if (!is64) - emit_zext_32_rd_t1(&rd, ctx); - emit(rv_bltu(rd, RV_REG_T1, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JGE | BPF_K: case BPF_JMP32 | BPF_JGE | BPF_K: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - emit_imm(RV_REG_T1, imm, ctx); - if (!is64) - emit_zext_32_rd_t1(&rd, ctx); - emit(rv_bgeu(rd, RV_REG_T1, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JLE | BPF_K: case BPF_JMP32 | BPF_JLE | BPF_K: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - emit_imm(RV_REG_T1, imm, ctx); - if (!is64) - emit_zext_32_rd_t1(&rd, ctx); - emit(rv_bgeu(RV_REG_T1, rd, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JNE | BPF_K: case BPF_JMP32 | BPF_JNE | BPF_K: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - emit_imm(RV_REG_T1, imm, ctx); - if (!is64) - emit_zext_32_rd_t1(&rd, ctx); - emit(rv_bne(rd, RV_REG_T1, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JSGT | BPF_K: case BPF_JMP32 | BPF_JSGT | BPF_K: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - emit_imm(RV_REG_T1, imm, ctx); - if (!is64) - emit_sext_32_rd(&rd, ctx); - emit(rv_blt(RV_REG_T1, rd, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JSLT | BPF_K: case BPF_JMP32 | BPF_JSLT | BPF_K: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - emit_imm(RV_REG_T1, imm, ctx); - if (!is64) - emit_sext_32_rd(&rd, ctx); - emit(rv_blt(rd, RV_REG_T1, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JSGE | BPF_K: case BPF_JMP32 | BPF_JSGE | BPF_K: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - emit_imm(RV_REG_T1, imm, ctx); - if (!is64) - emit_sext_32_rd(&rd, ctx); - emit(rv_bge(rd, RV_REG_T1, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JSLE | BPF_K: case BPF_JMP32 | BPF_JSLE | BPF_K: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; - emit_imm(RV_REG_T1, imm, ctx); - if (!is64) - emit_sext_32_rd(&rd, ctx); - emit(rv_bge(RV_REG_T1, rd, rvoff >> 1), ctx); - break; case BPF_JMP | BPF_JSET | BPF_K: case BPF_JMP32 | BPF_JSET | BPF_K: - if (rv_offset_check(&rvoff, off, i, ctx)) - return -1; + rvoff = rv_offset(i, off, ctx); + s = ctx->ninsns; emit_imm(RV_REG_T1, imm, ctx); - if (!is64) - emit_zext_32_rd_t1(&rd, ctx); - emit(rv_and(RV_REG_T1, rd, RV_REG_T1), ctx); - emit(rv_bne(RV_REG_T1, RV_REG_ZERO, rvoff >> 1), ctx); + if (!is64) { + if (is_signed_bpf_cond(BPF_OP(code))) + emit_sext_32_rd(&rd, ctx); + else + emit_zext_32_rd_t1(&rd, ctx); + } + e = ctx->ninsns; + + /* Adjust for extra insns */ + rvoff -= (e - s) << 2; + + if (BPF_OP(code) == BPF_JSET) { + /* Adjust for and */ + rvoff -= 4; + emit(rv_and(RV_REG_T1, rd, RV_REG_T1), ctx); + emit_branch(BPF_JNE, RV_REG_T1, RV_REG_ZERO, rvoff, + ctx); + } else { + emit_branch(BPF_OP(code), rd, RV_REG_T1, rvoff, ctx); + } break; /* function call */ @@ -1557,6 +1569,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) { bool tmp_blinded = false, extra_pass = false; struct bpf_prog *tmp, *orig_prog = prog; + int pass = 0, prev_ninsns = 0, i; struct rv_jit_data *jit_data; struct rv_jit_context *ctx; unsigned int image_size; @@ -1596,15 +1609,25 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) prog = orig_prog; goto out_offset; } + for (i = 0; i < prog->len; i++) { + prev_ninsns += 32; + ctx->offset[i] = prev_ninsns; + } - /* First pass generates the ctx->offset, but does not emit an image. */ - if (build_body(ctx, extra_pass)) { - prog = orig_prog; - goto out_offset; + for (i = 0; i < 16; i++) { + pass++; + ctx->ninsns = 0; + if (build_body(ctx, extra_pass)) { + prog = orig_prog; + goto out_offset; + } + build_prologue(ctx); + ctx->epilogue_offset = ctx->ninsns; + build_epilogue(ctx); + if (ctx->ninsns == prev_ninsns) + break; + prev_ninsns = ctx->ninsns; } - build_prologue(ctx); - ctx->epilogue_offset = ctx->ninsns; - build_epilogue(ctx); /* Allocate image, now that we know the size. */ image_size = sizeof(u32) * ctx->ninsns; @@ -1619,6 +1642,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) /* Second, real pass, that acutally emits the image. */ ctx->insns = (u32 *)jit_data->image; skip_init_ctx: + pass++; ctx->ninsns = 0; build_prologue(ctx); @@ -1630,7 +1654,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) build_epilogue(ctx); if (bpf_jit_enable > 1) - bpf_jit_dump(prog->len, image_size, 2, ctx->insns); + bpf_jit_dump(prog->len, image_size, pass, ctx->insns); prog->bpf_func = (void *)ctx->insns; prog->jited = 1; From patchwork Mon Dec 16 09:13:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 1210165 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="qIUfuvAf"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 47bwZ31sVcz9sQp for ; Mon, 16 Dec 2019 20:14:03 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727126AbfLPJOC (ORCPT ); Mon, 16 Dec 2019 04:14:02 -0500 Received: from mail-pf1-f193.google.com ([209.85.210.193]:32854 "EHLO mail-pf1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727114AbfLPJOB (ORCPT ); Mon, 16 Dec 2019 04:14:01 -0500 Received: by mail-pf1-f193.google.com with SMTP id y206so5259709pfb.0; Mon, 16 Dec 2019 01:14:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=rd0ylQkfUXD47lMr1COolacBtWcmBE98gO//niHtxXI=; b=qIUfuvAf5plY9AdGHbegpurFDTmDKuRlVF3Lz0ZeeaL8aWA07KSjFUz5a4g3OkJJl0 7sbcqLQyJc8jhoPW73hcA42kFw6TJBw8xoN2xOZ80LkhaZbBFFE1384G2YNKLYH7cLvl QjT5IkEKFLuNPoPbfMEzv/19Yt/x7PznYObaS7/cbRag8ccccqbCFYY2EhHR0mUtvjXX REtJ63n76pIK3I7W2la1dJE4GhvFaWJDifDnthNlm55cutG1eDhWBvphwKnzV23yo/7W gq5R4rR4ItY/qmIm48K0OthXZRU6AH2lqdbfKsN8cClVt9DIc9CB0VIrlfZxalpMo6lH daHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rd0ylQkfUXD47lMr1COolacBtWcmBE98gO//niHtxXI=; b=hnEzyrD7L3pzF4BilZuKH9vGLv2XLy7F8nEquKpRbfJoLRO2+/9jLWpr5Gh62H9rUi jE5NdX/36tLJ4k1zlsAWtcwq2rPsFVwxv5hjRhBKICnmpThbh9CadKWCmAmZUb6UVBgg bTkCkaxo9yqLYNeknvQIMcK4qankXecyTXA2/0h+EeL9E5ajQr8ZFBzJ5VrcwQXtecnr 9NECqh+gny57spvXNcdsYqfXndfKEZ2Uh1roHzvffI0W7JatyPU77tJE/Cv40hZtTZXp TysljVXghjA6QPR5FviXmC5DR2DMycIPR/taX8ugLax8P/dfOXWr3lXfAfHG0oUispSe mtPw== X-Gm-Message-State: APjAAAW0vPnuY9BMW3UxIYifR+8sceUZZ1XGF0OMYZvET0uGp/SPWoHt +OKz4tHN4JA62jPtDRyx2Ck= X-Google-Smtp-Source: APXvYqwvlSWpPBFZbhR83GOWOvhvx/HkAxTGzSWuJq1dD/CdoLWHqImIZK+iC3ThDTzoYJwhgDpBeg== X-Received: by 2002:a63:1f0c:: with SMTP id f12mr17305605pgf.247.1576487640533; Mon, 16 Dec 2019 01:14:00 -0800 (PST) Received: from btopel-mobl.ger.intel.com (fmdmzpr04-ext.fm.intel.com. [192.55.55.39]) by smtp.gmail.com with ESMTPSA id x21sm12505033pfn.164.2019.12.16.01.13.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Dec 2019 01:14:00 -0800 (PST) From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: daniel@iogearbox.net, ast@kernel.org, netdev@vger.kernel.org Cc: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , linux-riscv@lists.infradead.org, bpf@vger.kernel.org Subject: [PATCH bpf-next v2 3/9] riscv, bpf: add support for far branching when emitting tail call Date: Mon, 16 Dec 2019 10:13:37 +0100 Message-Id: <20191216091343.23260-4-bjorn.topel@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191216091343.23260-1-bjorn.topel@gmail.com> References: <20191216091343.23260-1-bjorn.topel@gmail.com> MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Start use the emit_branch() function in the tail call emitter in order to support far branching. Signed-off-by: Björn Töpel Reviewed-by: Palmer Dabbelt Acked-by: Palmer Dabbelt --- arch/riscv/net/bpf_jit_comp.c | 22 +++------------------- 1 file changed, 3 insertions(+), 19 deletions(-) diff --git a/arch/riscv/net/bpf_jit_comp.c b/arch/riscv/net/bpf_jit_comp.c index e599458a9bcd..c38c95df3440 100644 --- a/arch/riscv/net/bpf_jit_comp.c +++ b/arch/riscv/net/bpf_jit_comp.c @@ -496,16 +496,6 @@ static int is_12b_check(int off, int insn) return 0; } -static int is_13b_check(int off, int insn) -{ - if (!is_13b_int(off)) { - pr_err("bpf-jit: insn=%d 13b < offset=%d not supported yet!\n", - insn, (int)off); - return -1; - } - return 0; -} - static int is_21b_check(int off, int insn) { if (!is_21b_int(off)) { @@ -744,18 +734,14 @@ static int emit_bpf_tail_call(int insn, struct rv_jit_context *ctx) return -1; emit(rv_lwu(RV_REG_T1, off, RV_REG_A1), ctx); off = (tc_ninsn - (ctx->ninsns - start_insn)) << 2; - if (is_13b_check(off, insn)) - return -1; - emit(rv_bgeu(RV_REG_A2, RV_REG_T1, off >> 1), ctx); + emit_branch(BPF_JGE, RV_REG_A2, RV_REG_T1, off, ctx); /* if (--TCC < 0) * goto out; */ emit(rv_addi(RV_REG_T1, tcc, -1), ctx); off = (tc_ninsn - (ctx->ninsns - start_insn)) << 2; - if (is_13b_check(off, insn)) - return -1; - emit(rv_blt(RV_REG_T1, RV_REG_ZERO, off >> 1), ctx); + emit_branch(BPF_JSLT, RV_REG_T1, RV_REG_ZERO, off, ctx); /* prog = array->ptrs[index]; * if (!prog) @@ -768,9 +754,7 @@ static int emit_bpf_tail_call(int insn, struct rv_jit_context *ctx) return -1; emit(rv_ld(RV_REG_T2, off, RV_REG_T2), ctx); off = (tc_ninsn - (ctx->ninsns - start_insn)) << 2; - if (is_13b_check(off, insn)) - return -1; - emit(rv_beq(RV_REG_T2, RV_REG_ZERO, off >> 1), ctx); + emit_branch(BPF_JEQ, RV_REG_T2, RV_REG_ZERO, off, ctx); /* goto *(prog->bpf_func + 4); */ off = offsetof(struct bpf_prog, bpf_func); From patchwork Mon Dec 16 09:13:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 1210168 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="XPQz4evg"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 47bwZ60ZwGz9sQp for ; Mon, 16 Dec 2019 20:14:06 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727138AbfLPJOF (ORCPT ); Mon, 16 Dec 2019 04:14:05 -0500 Received: from mail-pl1-f194.google.com ([209.85.214.194]:34819 "EHLO mail-pl1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727086AbfLPJOE (ORCPT ); Mon, 16 Dec 2019 04:14:04 -0500 Received: by mail-pl1-f194.google.com with SMTP id g6so1574194plt.2; Mon, 16 Dec 2019 01:14:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=gJMg0Q8sOUthYx3cFqSRyerb3CBS/OU8cO8MoQc4pJM=; b=XPQz4evg0VbB6K1XDWp53Kbjm0ht98HVwNnW6mJQ1O+P9za2E0H0FBgIUI4lWGSRmm e47U7DRE05fU6Jm6GfUqHCUwTBjRBlkDzzGQkB/DDfc2zpuYuKZ51M9IYgRfKC/Vcb4E MbgHbf51mM7VdA+w/D8ywHwabtqnyU83pOt//xx4hy/5KAp87GRUUrFHEHxo6j6KtoQS VAaQOfZtMIdTLWUWXhSzLv/lZt6BI5JCeVpu/c6z2o0hdLLPZYw4+23KzAND93vM5VNH 1SKhLVSBAuozBMnV9W4mMBhf+lfFjwkaPGr4bUiqm9eg1I2Rbt51W+L15oMr8AXf+/Ou a9IA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gJMg0Q8sOUthYx3cFqSRyerb3CBS/OU8cO8MoQc4pJM=; b=jbbOViK6Ta8397wcr1RCeMHmMtVghSIKyPQlNFXqd2E9HyZowlqcwg8AQqXSkHmMvy AVFxmSS07pzjpUfJdmznvxTzkb/iSCbenREybOtLUBj2I0hew5aQkxCgw7WB/B2tDAnf /tGFlhnBbxblSbeNl891ItUMDMcjKpQ5G8FDj7qDD33texir5OFCDoOkmkSvu+rW12iG IsWfDlA7jDSIo5C8qHutVAvjPbNctQ1iXcnWJCO5VpjQFjPGntOoVtB4y780rSPiRoIm NNi4FzdajOagBelHM18EpczdhVhTWQv8aVMZ9rvw7mGI+3SSz5RD2m7BvdccWtjkjUn2 GVzQ== X-Gm-Message-State: APjAAAWyiRlmHiHK6a1Zw/mrEPuftm/n0BjCCNPFyLNt8UbcZcYXBzDH YLGo+4Vjp1q5sswDuF5cPZ8= X-Google-Smtp-Source: APXvYqxc8umv14WzkAH/BRhw9GggK6mAxe2lkuZid3gzCbuWbXlbEx2Br/c0kwVybokpglnbQOzepQ== X-Received: by 2002:a17:90b:85:: with SMTP id bb5mr16072485pjb.22.1576487643481; Mon, 16 Dec 2019 01:14:03 -0800 (PST) Received: from btopel-mobl.ger.intel.com (fmdmzpr04-ext.fm.intel.com. [192.55.55.39]) by smtp.gmail.com with ESMTPSA id x21sm12505033pfn.164.2019.12.16.01.14.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Dec 2019 01:14:03 -0800 (PST) From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: daniel@iogearbox.net, ast@kernel.org, netdev@vger.kernel.org Cc: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , linux-riscv@lists.infradead.org, bpf@vger.kernel.org, Luke Nelson , Xi Wang Subject: [PATCH bpf-next v2 4/9] riscv, bpf: add support for far jumps and exits Date: Mon, 16 Dec 2019 10:13:38 +0100 Message-Id: <20191216091343.23260-5-bjorn.topel@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191216091343.23260-1-bjorn.topel@gmail.com> References: <20191216091343.23260-1-bjorn.topel@gmail.com> MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org This commit add support for far (offset > 21b) jumps and exits. Reviewed-by: Luke Nelson Cc: Xi Wang Signed-off-by: Björn Töpel --- arch/riscv/net/bpf_jit_comp.c | 37 ++++++++++++++++------------------- 1 file changed, 17 insertions(+), 20 deletions(-) diff --git a/arch/riscv/net/bpf_jit_comp.c b/arch/riscv/net/bpf_jit_comp.c index c38c95df3440..2fc0f24ad30f 100644 --- a/arch/riscv/net/bpf_jit_comp.c +++ b/arch/riscv/net/bpf_jit_comp.c @@ -496,16 +496,6 @@ static int is_12b_check(int off, int insn) return 0; } -static int is_21b_check(int off, int insn) -{ - if (!is_21b_int(off)) { - pr_err("bpf-jit: insn=%d 21b < offset=%d not supported yet!\n", - insn, (int)off); - return -1; - } - return 0; -} - static void emit_imm(u8 rd, s64 val, struct rv_jit_context *ctx) { /* Note that the immediate from the add is sign-extended, @@ -820,6 +810,21 @@ static void emit_sext_32_rd(u8 *rd, struct rv_jit_context *ctx) *rd = RV_REG_T2; } +static void emit_jump_and_link(u8 rd, int rvoff, struct rv_jit_context *ctx) +{ + s64 upper, lower; + + if (is_21b_int(rvoff)) { + emit(rv_jal(rd, rvoff >> 1), ctx); + return; + } + + upper = (rvoff + (1 << 11)) >> 12; + lower = rvoff & 0xfff; + emit(rv_auipc(RV_REG_T1, upper), ctx); + emit(rv_jalr(rd, RV_REG_T1, lower), ctx); +} + static bool is_signed_bpf_cond(u8 cond) { return cond == BPF_JSGT || cond == BPF_JSLT || @@ -1101,13 +1106,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, /* JUMP off */ case BPF_JMP | BPF_JA: rvoff = rv_offset(i, off, ctx); - if (!is_21b_int(rvoff)) { - pr_err("bpf-jit: insn=%d offset=%d not supported yet!\n", - i, rvoff); - return -1; - } - - emit(rv_jal(RV_REG_ZERO, rvoff >> 1), ctx); + emit_jump_and_link(RV_REG_ZERO, rvoff, ctx); break; /* IF (dst COND src) JUMP off */ @@ -1245,9 +1244,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, break; rvoff = epilogue_offset(ctx); - if (is_21b_check(rvoff, i)) - return -1; - emit(rv_jal(RV_REG_ZERO, rvoff >> 1), ctx); + emit_jump_and_link(RV_REG_ZERO, rvoff, ctx); break; /* dst = imm64 */ From patchwork Mon Dec 16 09:13:39 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 1210170 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="vgYlauhS"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 47bwZ91Mrsz9sQp for ; Mon, 16 Dec 2019 20:14:09 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727151AbfLPJOI (ORCPT ); Mon, 16 Dec 2019 04:14:08 -0500 Received: from mail-pf1-f194.google.com ([209.85.210.194]:41079 "EHLO mail-pf1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727086AbfLPJOG (ORCPT ); Mon, 16 Dec 2019 04:14:06 -0500 Received: by mail-pf1-f194.google.com with SMTP id s18so5245685pfd.8; Mon, 16 Dec 2019 01:14:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+QLmugLg69uwoq+Jwozb0oyKbaFcKUgep+k5spKK228=; b=vgYlauhSVRfk3SG2Bd3rFWaU/N5IO+0XFjRzwLL4Nj2mEqZngDETSaWN91nNeMPO3C msXmmWZHLDuxE48AHdWqXT6qEl9CAAhWHMyiM1n7Nlruv3uqa7ZAcrKoQjZ9ZSPtCV40 1yGYGdpgJ2xFvUFfoQwM+ExMaNHpxQ07VJg3pCVLHTpgjsEME9vSonPXVDaGDeoHBbka vTqTq2ktbA86v0VieraSm+XBruaQNdvt91/NGAYuYphJzCbvmHl730QiqLx/6Hwu8muD OhyyU0s+3mjUYLfi0bwMKNe5LRBQQF5hki2gJMFFsVw3zPGyWvAmXNGTcL1mDAeHHf3F /2bQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+QLmugLg69uwoq+Jwozb0oyKbaFcKUgep+k5spKK228=; b=udUZ4maWiHfqH07Vd6HNjxjUZo15RjLjFqFNys/j9iuhaRCbF9sOppsWj2sMhHGUM/ tJEm2S7cVZWrPzvGTFJCnzEYAm4D6cceAxDw5TT8DEVgFxQZzj7e4jsDKrPekvRxjSd9 KancQuIs5WCHZBggbUyxLqhn56cPy7PvxdbD3Zyd3tTFu5G+bLWNA4hU/FsElmKQ7IX8 UyrV89Hy5jz4xH5OMN2a8jWOjwEf62XKNmI1UnmEmQu3AacX56vxi3ldji4py2jtVo4u z+PsFcJhUOVC+THP3CbO1cQb6b0C6n/wOeTS2UzK8Z7bS+1QhrT3d5pZN8/rdbvD1OCu OYXA== X-Gm-Message-State: APjAAAWJTSYn4olDvc18hN3aQGu1bnxY8Lp6+WbFkT5v58cmd3DZCeeP kihz/ZtKd844lH6/vvwTgVE= X-Google-Smtp-Source: APXvYqwNqrSXeV+vfLV2wS5LMUNUPbO2epEFOMwSUXehrzv1nnOdTQ7R3mn/CNh1KLbocAfAIqkurg== X-Received: by 2002:a63:590e:: with SMTP id n14mr16640396pgb.10.1576487645927; Mon, 16 Dec 2019 01:14:05 -0800 (PST) Received: from btopel-mobl.ger.intel.com (fmdmzpr04-ext.fm.intel.com. [192.55.55.39]) by smtp.gmail.com with ESMTPSA id x21sm12505033pfn.164.2019.12.16.01.14.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Dec 2019 01:14:05 -0800 (PST) From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: daniel@iogearbox.net, ast@kernel.org, netdev@vger.kernel.org Cc: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , linux-riscv@lists.infradead.org, bpf@vger.kernel.org Subject: [PATCH bpf-next v2 5/9] riscv, bpf: optimize BPF tail calls Date: Mon, 16 Dec 2019 10:13:39 +0100 Message-Id: <20191216091343.23260-6-bjorn.topel@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191216091343.23260-1-bjorn.topel@gmail.com> References: <20191216091343.23260-1-bjorn.topel@gmail.com> MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Remove one addi, and instead use the offset part of jalr. Signed-off-by: Björn Töpel Reviewed-by: Palmer Dabbelt Acked-by: Palmer Dabbelt --- arch/riscv/net/bpf_jit_comp.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/arch/riscv/net/bpf_jit_comp.c b/arch/riscv/net/bpf_jit_comp.c index 2fc0f24ad30f..8aa19c846881 100644 --- a/arch/riscv/net/bpf_jit_comp.c +++ b/arch/riscv/net/bpf_jit_comp.c @@ -552,7 +552,7 @@ static int epilogue_offset(struct rv_jit_context *ctx) return (to - from) << 2; } -static void __build_epilogue(u8 reg, struct rv_jit_context *ctx) +static void __build_epilogue(bool is_tail_call, struct rv_jit_context *ctx) { int stack_adjust = ctx->stack_size, store_offset = stack_adjust - 8; @@ -589,9 +589,11 @@ static void __build_epilogue(u8 reg, struct rv_jit_context *ctx) emit(rv_addi(RV_REG_SP, RV_REG_SP, stack_adjust), ctx); /* Set return value. */ - if (reg == RV_REG_RA) + if (!is_tail_call) emit(rv_addi(RV_REG_A0, RV_REG_A5, 0), ctx); - emit(rv_jalr(RV_REG_ZERO, reg, 0), ctx); + emit(rv_jalr(RV_REG_ZERO, is_tail_call ? RV_REG_T3 : RV_REG_RA, + is_tail_call ? 4 : 0), /* skip TCC init */ + ctx); } /* return -1 or inverted cond */ @@ -751,9 +753,8 @@ static int emit_bpf_tail_call(int insn, struct rv_jit_context *ctx) if (is_12b_check(off, insn)) return -1; emit(rv_ld(RV_REG_T3, off, RV_REG_T2), ctx); - emit(rv_addi(RV_REG_T3, RV_REG_T3, 4), ctx); emit(rv_addi(RV_REG_TCC, RV_REG_T1, 0), ctx); - __build_epilogue(RV_REG_T3, ctx); + __build_epilogue(true, ctx); return 0; } @@ -1504,7 +1505,7 @@ static void build_prologue(struct rv_jit_context *ctx) static void build_epilogue(struct rv_jit_context *ctx) { - __build_epilogue(RV_REG_RA, ctx); + __build_epilogue(false, ctx); } static int build_body(struct rv_jit_context *ctx, bool extra_pass) From patchwork Mon Dec 16 09:13:40 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 1210172 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="AoKOXKO/"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 47bwZC3XbKz9sRM for ; Mon, 16 Dec 2019 20:14:11 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727152AbfLPJOK (ORCPT ); Mon, 16 Dec 2019 04:14:10 -0500 Received: from mail-pf1-f193.google.com ([209.85.210.193]:44653 "EHLO mail-pf1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727086AbfLPJOJ (ORCPT ); Mon, 16 Dec 2019 04:14:09 -0500 Received: by mail-pf1-f193.google.com with SMTP id d199so5230109pfd.11; Mon, 16 Dec 2019 01:14:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=QO2XYproH0t5I47F5mhtBi9AMEJLuvvc53hQXLZNLWQ=; b=AoKOXKO/1ir4bT+D1DlgUf1isDtUDWHygosuKaYk74JMk0vJNwzkDiuuwuQ9fngA1n oYb73MSGF+jaWJ0xNyXkE3jFt6O6avV/NLsFX5CLRbXiXupSyDFL8eEjKRdY6tNTBZXp WYawDjkXM9XmDchvoTvdFZh9lgNhR5sHnoqZT/02LTG5dszcvAlzWP6dKegVdNbF+9NX yORcbNJ30Z9aWZLfWCleQxTYiGtFnBeJPQnNcpnFvi/zfjB11qmMKay7tQBLZ7C2ZDCZ kZ8WRsNgPuAH7N723ElFNiM5b8rNYxcLlOdf39fvw7um1NXRXMrzqtcdIH6p8kY6RNbE nYvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=QO2XYproH0t5I47F5mhtBi9AMEJLuvvc53hQXLZNLWQ=; b=H6vjoUlWqbSxlUN1AQ85+HMq9iSxytWSMtdKWjUhaf3iFNYx2MmleOWNilteX4PGE8 qU6zMkibIqROoYeFy0aWML5sCvecscZWtwjBQiYeo7Kp/p/RU+pm7gvynGs+pMsqG6Ee 60O6uoyRRzgktgmnxgLAROVSri+6be0gjmDEKAEljN5m/y3LHNC4AE6u9FVtf6JYlnBn gKlSxhsscIKx1zGM75tI6ZY9G8IC33q4eHCj1PTiM3v18b7tgR2fKrVNCfj0a2v4pSbi 81EQ79TCcrBQXfQB5ymDfb+2jJ7eX2TGWRsfaMjg7XqLl1quQAFSfeTvBcIfiDrcbCSw eYvw== X-Gm-Message-State: APjAAAXhkdcGGsD/BCcQ/IIwbePt6A2XRW5jPMgwlq1qcpDaL6D1ABM+ KH41lgyP4GyR/Es8WDCQxOA= X-Google-Smtp-Source: APXvYqxUgfQ3E9jefxz8vWiRidBaCpHKdCxu+LuYaqveP9Nh/uP6bKmQCD9Iwj5dJoI/m61t7uUpKg== X-Received: by 2002:a63:4d4c:: with SMTP id n12mr17283981pgl.212.1576487648452; Mon, 16 Dec 2019 01:14:08 -0800 (PST) Received: from btopel-mobl.ger.intel.com (fmdmzpr04-ext.fm.intel.com. [192.55.55.39]) by smtp.gmail.com with ESMTPSA id x21sm12505033pfn.164.2019.12.16.01.14.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Dec 2019 01:14:08 -0800 (PST) From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: daniel@iogearbox.net, ast@kernel.org, netdev@vger.kernel.org Cc: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , linux-riscv@lists.infradead.org, bpf@vger.kernel.org Subject: [PATCH bpf-next v2 6/9] riscv, bpf: provide RISC-V specific JIT image alloc/free Date: Mon, 16 Dec 2019 10:13:40 +0100 Message-Id: <20191216091343.23260-7-bjorn.topel@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191216091343.23260-1-bjorn.topel@gmail.com> References: <20191216091343.23260-1-bjorn.topel@gmail.com> MIME-Version: 1.0 Sender: bpf-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org This commit makes sure that the JIT images is kept close to the kernel text, so BPF calls can use relative calling with auipc/jalr or jal instead of loading the full 64-bit address and jalr. The BPF JIT image region is 128 MB before the kernel text. Signed-off-by: Björn Töpel Reviewed-by: Palmer Dabbelt Acked-by: Palmer Dabbelt --- arch/riscv/include/asm/pgtable.h | 4 ++++ arch/riscv/net/bpf_jit_comp.c | 13 +++++++++++++ 2 files changed, 17 insertions(+) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 7ff0ed4f292e..cc3f49415620 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -404,6 +404,10 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma, #define VMALLOC_END (PAGE_OFFSET - 1) #define VMALLOC_START (PAGE_OFFSET - VMALLOC_SIZE) +#define BPF_JIT_REGION_SIZE (SZ_128M) +#define BPF_JIT_REGION_START (PAGE_OFFSET - BPF_JIT_REGION_SIZE) +#define BPF_JIT_REGION_END (VMALLOC_END) + /* * Roughly size the vmemmap space to be large enough to fit enough * struct pages to map half the virtual address space. Then diff --git a/arch/riscv/net/bpf_jit_comp.c b/arch/riscv/net/bpf_jit_comp.c index 8aa19c846881..46cff093f526 100644 --- a/arch/riscv/net/bpf_jit_comp.c +++ b/arch/riscv/net/bpf_jit_comp.c @@ -1656,3 +1656,16 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) tmp : orig_prog); return prog; } + +void *bpf_jit_alloc_exec(unsigned long size) +{ + return __vmalloc_node_range(size, PAGE_SIZE, BPF_JIT_REGION_START, + BPF_JIT_REGION_END, GFP_KERNEL, + PAGE_KERNEL_EXEC, 0, NUMA_NO_NODE, + __builtin_return_address(0)); +} + +void bpf_jit_free_exec(void *addr) +{ + return vfree(addr); +} From patchwork Mon Dec 16 09:13:41 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 1210174 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="u8C9Hkmc"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 47bwZF5zldz9sRM for ; Mon, 16 Dec 2019 20:14:13 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727180AbfLPJON (ORCPT ); Mon, 16 Dec 2019 04:14:13 -0500 Received: from mail-pj1-f66.google.com ([209.85.216.66]:38648 "EHLO mail-pj1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727086AbfLPJOM (ORCPT ); Mon, 16 Dec 2019 04:14:12 -0500 Received: by mail-pj1-f66.google.com with SMTP id l4so2702302pjt.5; Mon, 16 Dec 2019 01:14:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fDxKy5+xWcbxsQCLEwvK2/ipDbDNasXEESlHOrWAL4M=; b=u8C9Hkmch6zgJB9Mga36IoCRDeJvXzqUoHzx8LOnbISGd4W3Gng24bK1CIkKkx+ayZ teMbIbjJweClsHJas6IsDCQOuwHJ5oGZc/CjgfjXKv7f4kZNWjxG1hkhMBykkrD8+E8t vbl/8Inmanaijpu1v4oV+wXyZDlxGXlfckw7ejvlDy6S/IC/qyRWtawsWXC0KJ42I3Uu xga7OBta8KWUrFq3SVbB5GkYfxv9vA07371d9//AcXY/dePrW+bSOea6dCQka64gXOlg DVWAKmK3TBMbh6dCDADKfW9CQUJjepEJSOy/CmO7FhXtJjq9qlh4r0J4+755LxHg3aIZ OGEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fDxKy5+xWcbxsQCLEwvK2/ipDbDNasXEESlHOrWAL4M=; b=sR0+/plUKSwEH97rBR95FSquZzw/WpBRP5ZiIYk5QPkkbGIQWNNnylhNsa+tJrmWOr 5Y7jTAwFBBYnTV3BWRw6ILfRlAjzy7rEJpMVpuCFquxqD7DRRHTY3KOR7oDTXJT4NWhb lfiDawSIyoIIn7wwMWsuMEfvyTrXK7GCkc7Wf5Uvqp+3+Xdoyx8f0VOPb2RzwKrZNN87 HIkqoKuXmc1GlvbIYSIivLPe0Vf4RcuZsWcidAWLoLt77HbAaAlo2Hukv0kXzIF/p7NB 0t2LwGajW5PojiWBaL7m+Ss5TpTuBYKWX6rpNoAtruY6to2M1YgTpzpqW/ygonvlOsgI EypA== X-Gm-Message-State: APjAAAUhXPwqn8Rokjj8+f4Z3QguJ18iklfmatH4b5WfkRprVE7na+WI pwtTAjoAz2LagbnIRDOgzJs= X-Google-Smtp-Source: APXvYqy4RRBfPnji3ZSdrSQ8+HXCy69F7PXRhowUO8OlRP12zL9gdt20Ak0hfOVAkVh+ql/ziFcfOw== X-Received: by 2002:a17:90a:bc05:: with SMTP id w5mr16386171pjr.64.1576487650999; Mon, 16 Dec 2019 01:14:10 -0800 (PST) Received: from btopel-mobl.ger.intel.com (fmdmzpr04-ext.fm.intel.com. [192.55.55.39]) by smtp.gmail.com with ESMTPSA id x21sm12505033pfn.164.2019.12.16.01.14.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Dec 2019 01:14:10 -0800 (PST) From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: daniel@iogearbox.net, ast@kernel.org, netdev@vger.kernel.org Cc: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , linux-riscv@lists.infradead.org, bpf@vger.kernel.org Subject: [PATCH bpf-next v2 7/9] riscv, bpf: optimize calls Date: Mon, 16 Dec 2019 10:13:41 +0100 Message-Id: <20191216091343.23260-8-bjorn.topel@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191216091343.23260-1-bjorn.topel@gmail.com> References: <20191216091343.23260-1-bjorn.topel@gmail.com> MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Instead of using emit_imm() and emit_jalr() which can expand to six instructions, start using jal or auipc+jalr. Signed-off-by: Björn Töpel --- arch/riscv/net/bpf_jit_comp.c | 101 +++++++++++++++++++++------------- 1 file changed, 64 insertions(+), 37 deletions(-) diff --git a/arch/riscv/net/bpf_jit_comp.c b/arch/riscv/net/bpf_jit_comp.c index 46cff093f526..8d7e3343a08c 100644 --- a/arch/riscv/net/bpf_jit_comp.c +++ b/arch/riscv/net/bpf_jit_comp.c @@ -811,11 +811,12 @@ static void emit_sext_32_rd(u8 *rd, struct rv_jit_context *ctx) *rd = RV_REG_T2; } -static void emit_jump_and_link(u8 rd, int rvoff, struct rv_jit_context *ctx) +static void emit_jump_and_link(u8 rd, s64 rvoff, bool force_jalr, + struct rv_jit_context *ctx) { s64 upper, lower; - if (is_21b_int(rvoff)) { + if (rvoff && is_21b_int(rvoff) && !force_jalr) { emit(rv_jal(rd, rvoff >> 1), ctx); return; } @@ -832,6 +833,28 @@ static bool is_signed_bpf_cond(u8 cond) cond == BPF_JSGE || cond == BPF_JSLE; } +static int emit_call(bool fixed, u64 addr, struct rv_jit_context *ctx) +{ + s64 off = 0; + u64 ip; + u8 rd; + + if (addr && ctx->insns) { + ip = (u64)(long)(ctx->insns + ctx->ninsns); + off = addr - ip; + if (!is_32b_int(off)) { + pr_err("bpf-jit: target call addr %pK is out of range\n", + (void *)addr); + return -ERANGE; + } + } + + emit_jump_and_link(RV_REG_RA, off, !fixed, ctx); + rd = bpf_to_rv_reg(BPF_REG_0, ctx); + emit(rv_addi(rd, RV_REG_A0, 0), ctx); + return 0; +} + static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, bool extra_pass) { @@ -1107,7 +1130,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, /* JUMP off */ case BPF_JMP | BPF_JA: rvoff = rv_offset(i, off, ctx); - emit_jump_and_link(RV_REG_ZERO, rvoff, ctx); + emit_jump_and_link(RV_REG_ZERO, rvoff, false, ctx); break; /* IF (dst COND src) JUMP off */ @@ -1209,7 +1232,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, case BPF_JMP | BPF_CALL: { bool fixed; - int i, ret; + int ret; u64 addr; mark_call(ctx); @@ -1217,20 +1240,9 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, &fixed); if (ret < 0) return ret; - if (fixed) { - emit_imm(RV_REG_T1, addr, ctx); - } else { - i = ctx->ninsns; - emit_imm(RV_REG_T1, addr, ctx); - for (i = ctx->ninsns - i; i < 8; i++) { - /* nop */ - emit(rv_addi(RV_REG_ZERO, RV_REG_ZERO, 0), - ctx); - } - } - emit(rv_jalr(RV_REG_RA, RV_REG_T1, 0), ctx); - rd = bpf_to_rv_reg(BPF_REG_0, ctx); - emit(rv_addi(rd, RV_REG_A0, 0), ctx); + ret = emit_call(fixed, addr, ctx); + if (ret) + return ret; break; } /* tail call */ @@ -1245,7 +1257,7 @@ static int emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, break; rvoff = epilogue_offset(ctx); - emit_jump_and_link(RV_REG_ZERO, rvoff, ctx); + emit_jump_and_link(RV_REG_ZERO, rvoff, false, ctx); break; /* dst = imm64 */ @@ -1508,7 +1520,7 @@ static void build_epilogue(struct rv_jit_context *ctx) __build_epilogue(false, ctx); } -static int build_body(struct rv_jit_context *ctx, bool extra_pass) +static int build_body(struct rv_jit_context *ctx, bool extra_pass, int *offset) { const struct bpf_prog *prog = ctx->prog; int i; @@ -1520,12 +1532,12 @@ static int build_body(struct rv_jit_context *ctx, bool extra_pass) ret = emit_insn(insn, ctx, extra_pass); if (ret > 0) { i++; - if (ctx->insns == NULL) - ctx->offset[i] = ctx->ninsns; + if (offset) + offset[i] = ctx->ninsns; continue; } - if (ctx->insns == NULL) - ctx->offset[i] = ctx->ninsns; + if (offset) + offset[i] = ctx->ninsns; if (ret) return ret; } @@ -1553,8 +1565,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) struct bpf_prog *tmp, *orig_prog = prog; int pass = 0, prev_ninsns = 0, i; struct rv_jit_data *jit_data; + unsigned int image_size = 0; struct rv_jit_context *ctx; - unsigned int image_size; if (!prog->jit_requested) return orig_prog; @@ -1599,36 +1611,51 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) for (i = 0; i < 16; i++) { pass++; ctx->ninsns = 0; - if (build_body(ctx, extra_pass)) { + if (build_body(ctx, extra_pass, ctx->offset)) { prog = orig_prog; goto out_offset; } build_prologue(ctx); ctx->epilogue_offset = ctx->ninsns; build_epilogue(ctx); - if (ctx->ninsns == prev_ninsns) - break; + + if (ctx->ninsns == prev_ninsns) { + if (jit_data->header) + break; + + image_size = sizeof(u32) * ctx->ninsns; + jit_data->header = + bpf_jit_binary_alloc(image_size, + &jit_data->image, + sizeof(u32), + bpf_fill_ill_insns); + if (!jit_data->header) { + prog = orig_prog; + goto out_offset; + } + + ctx->insns = (u32 *)jit_data->image; + /* Now, when the image is allocated, the image + * can potentially shrink more (auipc/jalr -> + * jal). + */ + } prev_ninsns = ctx->ninsns; } - /* Allocate image, now that we know the size. */ - image_size = sizeof(u32) * ctx->ninsns; - jit_data->header = bpf_jit_binary_alloc(image_size, &jit_data->image, - sizeof(u32), - bpf_fill_ill_insns); - if (!jit_data->header) { + if (i == 16) { + pr_err("bpf-jit: image did not converge in <%d passes!\n", i); + bpf_jit_binary_free(jit_data->header); prog = orig_prog; goto out_offset; } - /* Second, real pass, that acutally emits the image. */ - ctx->insns = (u32 *)jit_data->image; skip_init_ctx: pass++; ctx->ninsns = 0; build_prologue(ctx); - if (build_body(ctx, extra_pass)) { + if (build_body(ctx, extra_pass, NULL)) { bpf_jit_binary_free(jit_data->header); prog = orig_prog; goto out_offset; From patchwork Mon Dec 16 09:13:42 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 1210176 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="BckQdlsi"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 47bwZJ3Ndxz9sRM for ; Mon, 16 Dec 2019 20:14:16 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727114AbfLPJOP (ORCPT ); Mon, 16 Dec 2019 04:14:15 -0500 Received: from mail-pj1-f66.google.com ([209.85.216.66]:39514 "EHLO mail-pj1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727181AbfLPJOO (ORCPT ); Mon, 16 Dec 2019 04:14:14 -0500 Received: by mail-pj1-f66.google.com with SMTP id v93so2696058pjb.6; Mon, 16 Dec 2019 01:14:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=qigMm4EqiyANinCkFguk39Cl6HfqkkQUc4cgfeXf+Bk=; b=BckQdlsi9Dt8xH9pgrko9rHI855k7PWdnlZrhPLcQy1knWeDRCLkG4wmWQGrD3H9cl DTETfafa7A/GSj+Gt8KZfmw4HoHFjPhXHz/y7pWqTG0IGvA7h7Mx6PGpkYImGk/ItzsS bcVjB9HDw1w6xoHheeRZxgBzmmVjntrQWxIXykYzH3nGVQUFJzWRSz6wwW5JGRwxR+Jm pJ+ehcrStwAelXTioCm+lwQPBh9B+w34pAXtkbWD6vv8IKhncG2t/jDtjWKVKB93XLi9 OM4noU82f6UvjevNGfYkH9Ts/L4ck85HkMgQVjH6cKPjiizKKGgBVO5hpaNT62ZyDdsd nHZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=qigMm4EqiyANinCkFguk39Cl6HfqkkQUc4cgfeXf+Bk=; b=VlEn+FCqau5FdhluXgObc8j82OTYkTL1c6IzTVRtM8MBGQ1Q4dtr7i18R1vptCFeeY 89ctWOJyWur9sil+70vVrZ4Wkt4jzsEe2zFReNlVsLt1mvIxtmmfTvEahZmnJUIBKTuH Y+vwZg78nW48d41hy0Od5g357eYSwI9x3e72SsJqt4rrTwuChHDYLUQpT8yCJlp6Zv0F mgIHnxcxi6ffruf5FpPrHXdi/u2geT/d8XdhSqEEeeoQRL3ZlMntOf/zZ0U0WCdgLRJ3 XnGkUoMcFQBeMnwn1dv0kRlUia04Aba5UKMsoF4vJaWZPtozQwwz0852A4hrdDRIeUf1 Zavg== X-Gm-Message-State: APjAAAUj9VA64IuDnIo9M6EMcMaIHIIWB9xrkNBBXg21r2dnV5QZc/xe CQMmjTv/mydfANeA23/a28k= X-Google-Smtp-Source: APXvYqxYqrVllDXaRnxxFZ6SRKc2DpVooTXs1XkSRJUyLVLcEMZa21+AMT6rkirvRyXtPMIQO7C7OQ== X-Received: by 2002:a17:90b:30c4:: with SMTP id hi4mr16752987pjb.62.1576487653507; Mon, 16 Dec 2019 01:14:13 -0800 (PST) Received: from btopel-mobl.ger.intel.com (fmdmzpr04-ext.fm.intel.com. [192.55.55.39]) by smtp.gmail.com with ESMTPSA id x21sm12505033pfn.164.2019.12.16.01.14.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Dec 2019 01:14:13 -0800 (PST) From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: daniel@iogearbox.net, ast@kernel.org, netdev@vger.kernel.org Cc: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , linux-riscv@lists.infradead.org, bpf@vger.kernel.org Subject: [PATCH bpf-next v2 8/9] riscv, bpf: add missing uapi header for BPF_PROG_TYPE_PERF_EVENT programs Date: Mon, 16 Dec 2019 10:13:42 +0100 Message-Id: <20191216091343.23260-9-bjorn.topel@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191216091343.23260-1-bjorn.topel@gmail.com> References: <20191216091343.23260-1-bjorn.topel@gmail.com> MIME-Version: 1.0 Sender: bpf-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org Add missing uapi header the BPF_PROG_TYPE_PERF_EVENT programs by exporting struct user_regs_struct instead of struct pt_regs which is in-kernel only. Signed-off-by: Björn Töpel Reviewed-by: Palmer Dabbelt Acked-by: Palmer Dabbelt --- arch/riscv/include/uapi/asm/bpf_perf_event.h | 9 +++++++++ tools/include/uapi/asm/bpf_perf_event.h | 2 ++ 2 files changed, 11 insertions(+) create mode 100644 arch/riscv/include/uapi/asm/bpf_perf_event.h diff --git a/arch/riscv/include/uapi/asm/bpf_perf_event.h b/arch/riscv/include/uapi/asm/bpf_perf_event.h new file mode 100644 index 000000000000..6cb1c2823288 --- /dev/null +++ b/arch/riscv/include/uapi/asm/bpf_perf_event.h @@ -0,0 +1,9 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +#ifndef _UAPI__ASM_BPF_PERF_EVENT_H__ +#define _UAPI__ASM_BPF_PERF_EVENT_H__ + +#include + +typedef struct user_regs_struct bpf_user_pt_regs_t; + +#endif /* _UAPI__ASM_BPF_PERF_EVENT_H__ */ diff --git a/tools/include/uapi/asm/bpf_perf_event.h b/tools/include/uapi/asm/bpf_perf_event.h index 13a58531e6fa..39acc149d843 100644 --- a/tools/include/uapi/asm/bpf_perf_event.h +++ b/tools/include/uapi/asm/bpf_perf_event.h @@ -2,6 +2,8 @@ #include "../../arch/arm64/include/uapi/asm/bpf_perf_event.h" #elif defined(__s390__) #include "../../arch/s390/include/uapi/asm/bpf_perf_event.h" +#elif defined(__riscv) +#include "../../arch/riscv/include/uapi/asm/bpf_perf_event.h" #else #include #endif From patchwork Mon Dec 16 09:13:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 1210178 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="AEIfRIfu"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 47bwZM2K1Kz9sRM for ; Mon, 16 Dec 2019 20:14:19 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727181AbfLPJOR (ORCPT ); Mon, 16 Dec 2019 04:14:17 -0500 Received: from mail-pg1-f194.google.com ([209.85.215.194]:45006 "EHLO mail-pg1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727189AbfLPJOQ (ORCPT ); Mon, 16 Dec 2019 04:14:16 -0500 Received: by mail-pg1-f194.google.com with SMTP id x7so3308277pgl.11; Mon, 16 Dec 2019 01:14:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JJx1gb7dyIux/wvtlAjUM4Ul67VDqBxG4kjHz1T/apg=; b=AEIfRIfuKJKYD+qvWQaSBXL7IDzElcWqewDLhnppNot8arXsdKKANjssizbJgnCmKV SDIbaGEkWHjZoNKopu6kC3hbpLdvJngOZY92hPXZGZTMuJhbwyr7DOFHpkYU/Bly6t0J MgJeIFQ1I4RJl9criTK7V/4yXUSG7Hu01jTMp0JstURGvI+v2tpJYQ902utvaQEEZReq umZ+iVILzopn2A5DzvLjf11+vEJvzZPxY3t1i6Ev1oLJzj44HXNhysK65vc3HI1SZSS/ FZ6dh97VkM4iHm7mY34w1mQ0eOZzNFY+RbahdFWiY5ogJJeLCpxuc16JwcM8CNzn3EAM uG+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JJx1gb7dyIux/wvtlAjUM4Ul67VDqBxG4kjHz1T/apg=; b=oRb/T7Guhf9C+pNax6Ykpp4PRsHq/p1VSUQNshj9CuC+qoXR2/z+hQnYnPbDOFgjx4 v0HDlBDEnl/gkc5IVGjVwegwZOgwgTdugBRj7vTkoh/LZtUd/R3TymaYIGB+40UI4XgE ZJrldqxVQCKDZvIC3enVmC+YuSZN16SGvAq6rl/oS7IvvPU9EOTX2lgmlTKrR9RcMGK5 JjIRENyyUDbk6AYkZLiNg2a8aBCEIh0DI6ONp4jf8/kqs2uatMlmXRaLxADzaQEWhABo yP1pXD+gfy1OCdVALyzroP75Zv46jZtmpNhc7JLBUSqqmCTZMgHumpZDnXdAyk9d9bza FezA== X-Gm-Message-State: APjAAAVbi3vzeCPrX4j1TaEnABU3l01aMjeIctpdM2ab9m1nGY46npQJ 48tFWaW6tJkHGVTnie82xesNRVzS86I= X-Google-Smtp-Source: APXvYqxFeC9Wz/Ld4lIscKeMf0jMlKJu/7MJXqqy6/vQF89UWfwcx6tFmnqaY4nKmOGM+XOBrCNyHw== X-Received: by 2002:a62:7b54:: with SMTP id w81mr14816500pfc.127.1576487655949; Mon, 16 Dec 2019 01:14:15 -0800 (PST) Received: from btopel-mobl.ger.intel.com (fmdmzpr04-ext.fm.intel.com. [192.55.55.39]) by smtp.gmail.com with ESMTPSA id x21sm12505033pfn.164.2019.12.16.01.14.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 16 Dec 2019 01:14:15 -0800 (PST) From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: daniel@iogearbox.net, ast@kernel.org, netdev@vger.kernel.org Cc: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , linux-riscv@lists.infradead.org, bpf@vger.kernel.org Subject: [PATCH bpf-next v2 9/9] riscv, perf: add arch specific perf_arch_bpf_user_pt_regs Date: Mon, 16 Dec 2019 10:13:43 +0100 Message-Id: <20191216091343.23260-10-bjorn.topel@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191216091343.23260-1-bjorn.topel@gmail.com> References: <20191216091343.23260-1-bjorn.topel@gmail.com> MIME-Version: 1.0 Sender: bpf-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org RISC-V was missing a proper perf_arch_bpf_user_pt_regs macro for CONFIG_PERF_EVENT builds. Signed-off-by: Björn Töpel Reviewed-by: Palmer Dabbelt Acked-by: Palmer Dabbelt --- arch/riscv/include/asm/perf_event.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/arch/riscv/include/asm/perf_event.h b/arch/riscv/include/asm/perf_event.h index aefbfaa6a781..0234048b12bc 100644 --- a/arch/riscv/include/asm/perf_event.h +++ b/arch/riscv/include/asm/perf_event.h @@ -82,4 +82,8 @@ struct riscv_pmu { int irq; }; +#ifdef CONFIG_PERF_EVENTS +#define perf_arch_bpf_user_pt_regs(regs) (struct user_regs_struct *)regs +#endif + #endif /* _ASM_RISCV_PERF_EVENT_H */