From patchwork Sat Jan 19 18:06:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jakub Kicinski X-Patchwork-Id: 1027996 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=none (p=none dis=none) header.from=netronome.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=netronome-com.20150623.gappssmtp.com header.i=@netronome-com.20150623.gappssmtp.com header.b="YazRAnto"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 43hm4D3N6kz9s55 for ; Sun, 20 Jan 2019 05:07:24 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728970AbfASSHS (ORCPT ); Sat, 19 Jan 2019 13:07:18 -0500 Received: from mail-qk1-f196.google.com ([209.85.222.196]:36872 "EHLO mail-qk1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728903AbfASSHQ (ORCPT ); Sat, 19 Jan 2019 13:07:16 -0500 Received: by mail-qk1-f196.google.com with SMTP id g125so9885062qke.4 for ; Sat, 19 Jan 2019 10:07:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=netronome-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=g5TJa3/5mttYfIv5cSaIA23jtXJbLlswg/w/8Kjyu7g=; b=YazRAntoZ9ewIWwb/nx5ljo9nbD4lc767Lo4EsXXiElLdlKLmk8gioiGALDh1Fwl62 wMfHqhJ/wsc1Gfo6HX4aynEbUbLunvBZUpMK4keRuMhy04nVmvnasAfvC3mfsLhhmUp/ U6jCawA+rB3HAOve+AQ0MwcxErkPl9TfKsOtibWq7CtGBDgFiFovLEcLt82ysOoYBnDX abVF/uCvzSgBEUebmyzxDeMZriSNiogLJRE1G0R9vFf1JjKMsz60bkn3nO350eC1jQmF lS6Yt60p4Gg5iAxSJo+ZUnbBpO9dMnLFU35N/xeHWoYDwRkS7qo7e7HJEvtikeShmuuO SodA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=g5TJa3/5mttYfIv5cSaIA23jtXJbLlswg/w/8Kjyu7g=; b=OivE5uc1jB2xg4v5kllsxAlfdxc030Os02F7yfG0oBOoIhrFt9rmzEWyxcJcRn02ZB L4Kt9AMGIfAiOAyoza2DC6Q8JIkJz9dFh+5S46yqzkVxI7z07k07Ai1/gau74hWmAGK1 Q1dYrkbz51BSUaY4doE4oHPBE1/vm4bQb9bSoCkpQ+klUMYjs6imluSUgtIC9WKpHD3t kx7tw3hgtncUYRJaF9x8JubGUqF0OrgFCLJIHe8jd5kvMvgNH+r2/a9DiIpH89h/vKeA ECfwEKnE6uUC1hk3ZOu1wVczgmdH52tyuKtXTwDzxJfITKx6gwkrKVhNf2apRBoespgW P/IQ== X-Gm-Message-State: AJcUukcUijEEvIiWg6HEBus3Rq9FCNRL4bMAzjUpRj/2OGYn28LLnVTx 26nhFAsW29qQkVK7/4+8/FIjXw== X-Google-Smtp-Source: ALg8bN4UWBLzPKfXZgPsjw9Ekm8xanBV+Kf3RK/wRmvDNcSE4i1L8C4p+blZrfR6sA9v4Y/G+R1CcQ== X-Received: by 2002:a37:3692:: with SMTP id d140mr18810432qka.108.1547921235215; Sat, 19 Jan 2019 10:07:15 -0800 (PST) Received: from jkicinski-Precision-T1700.netronome.com ([66.60.152.14]) by smtp.gmail.com with ESMTPSA id v2sm46521518qte.75.2019.01.19.10.07.13 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 19 Jan 2019 10:07:14 -0800 (PST) From: Jakub Kicinski To: alexei.starovoitov@gmail.com, daniel@iogearbox.net Cc: netdev@vger.kernel.org, oss-drivers@netronome.com, yhs@fb.com, Jakub Kicinski Subject: [PATCH bpf-next v5 02/12] bpf: verifier: hard wire branches to dead code Date: Sat, 19 Jan 2019 10:06:46 -0800 Message-Id: <20190119180656.24635-3-jakub.kicinski@netronome.com> X-Mailer: git-send-email 2.19.2 In-Reply-To: <20190119180656.24635-1-jakub.kicinski@netronome.com> References: <20190119180656.24635-1-jakub.kicinski@netronome.com> MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Loading programs with dead code becomes more and more common, as people begin to patch constants at load time. Turn conditional jumps to unconditional ones, to avoid potential branch misprediction penalty. This optimization is enabled for privileged users only. For branches which just fall through we could just mark them as not seen and have dead code removal take care of them, but that seems less clean. v0.2: - don't call capable(CAP_SYS_ADMIN) twice (Jiong). v3: - fix GCC warning; Signed-off-by: Jakub Kicinski Acked-by: Yonghong Song --- kernel/bpf/verifier.c | 45 +++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 43 insertions(+), 2 deletions(-) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 56674a7c3778..71949c163b7a 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -6457,6 +6457,40 @@ static void sanitize_dead_code(struct bpf_verifier_env *env) } } +static bool insn_is_cond_jump(u8 code) +{ + u8 op; + + if (BPF_CLASS(code) != BPF_JMP) + return false; + + op = BPF_OP(code); + return op != BPF_JA && op != BPF_EXIT && op != BPF_CALL; +} + +static void opt_hard_wire_dead_code_branches(struct bpf_verifier_env *env) +{ + struct bpf_insn_aux_data *aux_data = env->insn_aux_data; + struct bpf_insn ja = BPF_JMP_IMM(BPF_JA, 0, 0, 0); + struct bpf_insn *insn = env->prog->insnsi; + const int insn_cnt = env->prog->len; + int i; + + for (i = 0; i < insn_cnt; i++, insn++) { + if (!insn_is_cond_jump(insn->code)) + continue; + + if (!aux_data[i + 1].seen) + ja.off = insn->off; + else if (!aux_data[i + 1 + insn->off].seen) + ja.off = 0; + else + continue; + + memcpy(insn, &ja, sizeof(ja)); + } +} + /* convert load instructions that access fields of a context type into a * sequence of instructions that access fields of the underlying structure: * struct __sk_buff -> struct sk_buff @@ -7148,6 +7182,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, struct bpf_verifier_env *env; struct bpf_verifier_log *log; int ret = -EINVAL; + bool is_priv; /* no program is valid */ if (ARRAY_SIZE(bpf_verifier_ops) == 0) @@ -7194,6 +7229,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, if (attr->prog_flags & BPF_F_ANY_ALIGNMENT) env->strict_alignment = false; + is_priv = capable(CAP_SYS_ADMIN); + env->allow_ptr_leaks = is_priv; + ret = replace_map_fd_with_map_ptr(env); if (ret < 0) goto skip_full_check; @@ -7211,8 +7249,6 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, if (!env->explored_states) goto skip_full_check; - env->allow_ptr_leaks = capable(CAP_SYS_ADMIN); - ret = check_subprogs(env); if (ret < 0) goto skip_full_check; @@ -7242,6 +7278,11 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, ret = check_max_stack_depth(env); /* instruction rewrites happen after this point */ + if (is_priv) { + if (ret == 0) + opt_hard_wire_dead_code_branches(env); + } + if (ret == 0) sanitize_dead_code(env);