From patchwork Tue Mar 26 18:05:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jiong Wang X-Patchwork-Id: 1065889 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="RoDLjVv8"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 44TJx336zJz9sT1 for ; Wed, 27 Mar 2019 05:06:47 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1732505AbfCZSGq (ORCPT ); Tue, 26 Mar 2019 14:06:46 -0400 Received: from mail-wm1-f65.google.com ([209.85.128.65]:51200 "EHLO mail-wm1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1732387AbfCZSGp (ORCPT ); Tue, 26 Mar 2019 14:06:45 -0400 Received: by mail-wm1-f65.google.com with SMTP id 4so13674533wmf.1 for ; Tue, 26 Mar 2019 11:06:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=mudG8RvVsPscJFtOW94defIRDtY8/IaEfskGfsPcl3o=; b=RoDLjVv8hYngagCtjx7KodJXiPKWLHLjPxn6Ue9w0uV8JG5JbQC9nyYQL8KandEiI1 XhfE0FjIlaCr7VkwdnEO8RNg/nOcixhAlc7nUEWLw4xzOgIJ/lxNye4/jps2PGfYW+4C NB9uxTHIlSTKdNrhjbxN0Otb5E25D+6ZzWIVVKVgyNPyKdkunkg3wUIx/QYpK4qHdz7L se2Mb5Tp5t314VzT1pehz/Ju5vBx0bysaRj8GrzhV4EnkHx8nDhBWpA1jmHLhW55L6Ov JoW19gXt8I1J10lxGPiLmJTQsne5Wxq6IBEcKAFKpF+g7UrEPif6v6//dByRiHH+Skdy 9rSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=mudG8RvVsPscJFtOW94defIRDtY8/IaEfskGfsPcl3o=; b=YxVkEI6tNsyqacRiDgJv6vHvWNtGVmphVwIsp1a+yqYbVBNqmN55TdUX+7YjFJiLGG dC6yJA9iMvjIm7Gni4butrPi00EmOoFD+HbSMq7aCUuOugeKWHF+1+HxFB9L1wZWQ6Eo xCeC/DKUBGfidDFEvHto8ztan2S7I8mu9vaO8eXkElIrgWXO/ZjfSDlBZle4z8WvCKgW FIsf9FaYvj0NQ6ylu+pNeZD6AkJyr3zE88JGIPRZbrG1Vbwqe7NbvbaGU3nmGyMmlAHx jYrzR0DDHW4JThNYg99gXcAJX+soNGGs9lOvuDtq+/52DxjVh06d/A91N09FKn6hf9hd yJqQ== X-Gm-Message-State: APjAAAUonJwezwAtdWUq52Mkpvohthx2OWKgi5HWcfxKhIJITU5EBsU4 ZaMmmblYizCUvTYBVwcjLqsmKA== X-Google-Smtp-Source: APXvYqx7hzQuy2XOs/tEbBMoZ7LRcdI0dA/dKXAdi6lHD2QbOkBU5NjtCmAh6aKpIGVvTJCGp4AD5A== X-Received: by 2002:a1c:3845:: with SMTP id f66mr10324433wma.41.1553623603801; Tue, 26 Mar 2019 11:06:43 -0700 (PDT) Received: from cbtest28.netronome.com ([217.38.71.146]) by smtp.gmail.com with ESMTPSA id i28sm36697534wrc.32.2019.03.26.11.06.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 26 Mar 2019 11:06:42 -0700 (PDT) From: Jiong Wang To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: bpf@vger.kernel.org, netdev@vger.kernel.org, oss-drivers@netronome.com, Jiong Wang Subject: [PATCH/RFC bpf-next 07/16] bpf: insert explicit zero extension instructions when bpf_jit_32bit_opt is true Date: Tue, 26 Mar 2019 18:05:30 +0000 Message-Id: <1553623539-15474-8-git-send-email-jiong.wang@netronome.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1553623539-15474-1-git-send-email-jiong.wang@netronome.com> References: <1553623539-15474-1-git-send-email-jiong.wang@netronome.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org This patch implements the zero extension insertion pass using bpf_patch_insn_data infrastructure. Once zero extensions are inserted, tell JIT back-ends about this through the new field env boolean "no_verifier_zext". We need this because user could enable or disable the insertion pass as they like through sysctl variable. Reviewed-by: Jakub Kicinski Signed-off-by: Jiong Wang --- include/linux/bpf.h | 1 + kernel/bpf/verifier.c | 45 ++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 45 insertions(+), 1 deletion(-) diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 5616a58..3336f93 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -359,6 +359,7 @@ struct bpf_prog_aux { u32 id; u32 func_cnt; /* used by non-func prog as the number of func progs */ u32 func_idx; /* 0 for non-func prog, the index in func array for func prog */ + bool no_verifier_zext; /* No zero extension insertion by verifier. */ bool offload_requested; struct bpf_prog **func; void *jit_data; /* JIT specific data. arch dependent */ diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 83448bb..57db451 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -7224,6 +7224,38 @@ static int opt_remove_nops(struct bpf_verifier_env *env) return 0; } +static int opt_subreg_zext(struct bpf_verifier_env *env) +{ + struct bpf_insn_aux_data *aux = env->insn_aux_data; + int i, delta = 0, len = env->prog->len; + struct bpf_insn *insns = env->prog->insnsi; + struct bpf_insn zext_patch[3]; + struct bpf_prog *new_prog; + + zext_patch[1] = BPF_ALU64_IMM(BPF_LSH, 0, 32); + zext_patch[2] = BPF_ALU64_IMM(BPF_RSH, 0, 32); + for (i = 0; i < len; i++) { + struct bpf_insn insn; + + if (!aux[i + delta].zext_dst) + continue; + + insn = insns[i + delta]; + zext_patch[0] = insn; + zext_patch[1].dst_reg = insn.dst_reg; + zext_patch[2].dst_reg = insn.dst_reg; + new_prog = bpf_patch_insn_data(env, i + delta, zext_patch, 3); + if (!new_prog) + return -ENOMEM; + env->prog = new_prog; + insns = new_prog->insnsi; + aux = env->insn_aux_data; + delta += 2; + } + + return 0; +} + /* convert load instructions that access fields of a context type into a * sequence of instructions that access fields of the underlying structure: * struct __sk_buff -> struct sk_buff @@ -8022,7 +8054,18 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, if (ret == 0) ret = check_max_stack_depth(env); - /* instruction rewrites happen after this point */ + /* Instruction rewrites happen after this point. + * For offload target, finalize hook has all aux insn info, do any + * customized work there. + */ + if (ret == 0 && bpf_jit_32bit_opt && + !bpf_prog_is_dev_bound(env->prog->aux)) { + ret = opt_subreg_zext(env); + env->prog->aux->no_verifier_zext = !!ret; + } else { + env->prog->aux->no_verifier_zext = true; + } + if (is_priv) { if (ret == 0) opt_hard_wire_dead_code_branches(env);