From patchwork Tue Jun 9 17:26:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 1306177 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=abdFKGls; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 49hH9J6gGRz9sQx for ; Wed, 10 Jun 2020 03:26:44 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731722AbgFIR0o (ORCPT ); Tue, 9 Jun 2020 13:26:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37780 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726938AbgFIR0m (ORCPT ); Tue, 9 Jun 2020 13:26:42 -0400 Received: from mail-pl1-x644.google.com (mail-pl1-x644.google.com [IPv6:2607:f8b0:4864:20::644]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2E8ECC05BD1E; Tue, 9 Jun 2020 10:26:42 -0700 (PDT) Received: by mail-pl1-x644.google.com with SMTP id n9so8312692plk.1; Tue, 09 Jun 2020 10:26:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=xdiuyaT4q5ifNZgwkk2eoy6a+eyBMSIgKVWGTekNeNI=; b=abdFKGlsVSDGF7HKzBIdGCAbqWpxQbcToJGapD2DGn4SvXoaqIA9HobdTvtxaM6WpX HNJGlcLoyRvtm9cCspC4awzoRUMnRRjGqTa1P90A+6NGeL6nZSLyawsObKwYXHuBhMnI HNvcGyHIho1qtAkXjUqyhljzImalb3ernVzJQBdmOJ1GxVITbNiOWy5V6NbL3PHW7KcR DwdMY0fZHGuwfxannCpVnmZ/IpP4idbbMd8R0XAGdLDRkMqVx3XsNDMboclLjYMTEzTG MTxS/D0fDOhTLiSIO9zAKuBvpHNIdiqVK3upql6H9D8/ilGXVUZQmJ4LgYR4kbL4v8KV aTjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xdiuyaT4q5ifNZgwkk2eoy6a+eyBMSIgKVWGTekNeNI=; b=t3CXpcJz/ICyh0ryMZyqQa7ObjvMojdNN1foCJakQcRRcfNY18NtOYPWAYQLZmz/9t C7xNp24Ig0RE9DanUkOaVr3SIVb3R5coywVYqR63PuSfJ3PYnU2SX2Cpa1/pPKbiFlPC VPWpZCEEmC11U8salhZBqTPRty5NB0qqvPjSX2wUTn46Gtg8Q9Uu9Y59E01S68K/stfv T4ir5/eZW342LL+K+YaNIw8ym+pbLurCeQTuiW80QlMMOz1QvkmVyevfrCGmzjRGC2rQ qy/J32B+pRecnoNDeoHBsJrKBOLPG1IpfgFq6u7l+pJKESn2wnI9B359+RzQjL7sVGR+ Fiig== X-Gm-Message-State: AOAM533lBQBB45hEkqAl74QOICLuHp8oFhWamcIekEOUYR4VpnuYpPED y/3Fq24S23IEXRo/BXo+Kvc= X-Google-Smtp-Source: ABdhPJwpB0ergjxQdCAtFDzwLdPg9G0RMr3TWW+1niPB5PmV37mAljEBYSAnWROjMsvVs2qe2jzJWA== X-Received: by 2002:a17:90a:7401:: with SMTP id a1mr5683557pjg.218.1591723601635; Tue, 09 Jun 2020 10:26:41 -0700 (PDT) Received: from btopel-mobl.ger.intel.com ([192.55.55.41]) by smtp.gmail.com with ESMTPSA id w190sm10390387pfw.35.2020.06.09.10.26.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Jun 2020 10:26:40 -0700 (PDT) From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: ast@kernel.org, daniel@iogearbox.net, bpf@vger.kernel.org, john.fastabend@gmail.com, toke@redhat.com Cc: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , magnus.karlsson@intel.com, netdev@vger.kernel.org, brouer@redhat.com, maciej.fijalkowski@intel.com Subject: [RFC PATCH bpf-next 1/2] bpf, xdp: add naive bpf_redirect_map() tail call detection Date: Tue, 9 Jun 2020 19:26:21 +0200 Message-Id: <20200609172622.37990-2-bjorn.topel@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200609172622.37990-1-bjorn.topel@gmail.com> References: <20200609172622.37990-1-bjorn.topel@gmail.com> MIME-Version: 1.0 Sender: bpf-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org From: Björn Töpel The bpf_redirect_map() BPF helper is used to redirect a packet to the endpoint referenced by a map element. Currently, there is no restrictment how the helper is called, e.g. ret = bpf_redirect_map(...); ... ret = bpf_redirect_map(...); ... // e.g. modify packet return ret; is a valid, albeit weird, program. The unconstrained nature of BPF (redirect) helpers has implications for the implementation of the XDP_REDIRECT call. Today, the redirect flow for a packet is along the lines of: for_each_packet() { act = bpf_prog_run_xdp(xdp_prog, xdp); // does bpf_redirect_map() call if (act == XDP_REDIRECT) ret = xdp_do_redirect(...); } xdp_do_flush(); The bpf_redirect_map() helper populates a per-cpu structure, which is picked up by the xdp_do_redirect() function that also performs the redirect action. What if the verifier could detect that certain programs have certain constraints, so the functiontionality performed by xdp_do_redirect() could be done directly from bpf_redirect_map()? Here, the verifier is taught to detect program where all bpf_redirect_map() calls, are tail calls. If a function call is a last action performed in another function, it is said to be a tail call -- not to be confused with the BPF tail call helper. This implementation is just a PoC, and not complete. It manages to detect tail calls by a naive peephole scheme. For the following program (the built in AF_XDP libbpf program): ... call bpf_redirect_map if w0 != 0 goto pc+X (goto exit) ... call bpf_redirect_map exit the verifier detects that all calls are tail calls. TODO: * Stateful tail call detection, instead of the current peephole one. * Add all helpers capable of returning XDP_REDIRECT, and not just bpf_redirect_map(). Signed-off-by: Björn Töpel --- include/linux/bpf_verifier.h | 2 ++ include/linux/filter.h | 19 +++++++++++- kernel/bpf/verifier.c | 53 +++++++++++++++++++++++++++++++++ net/core/filter.c | 57 ++++++++++++++++++++++++++++++++++++ 4 files changed, 130 insertions(+), 1 deletion(-) diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index ca08db4ffb5f..e41fd9264c94 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -411,6 +411,8 @@ struct bpf_verifier_env { u32 peak_states; /* longest register parentage chain walked for liveness marking */ u32 longest_mark_read_walk; + bool all_redirect_map_tail; + u32 redirect_map_call_cnt; }; __printf(2, 0) void bpf_verifier_vlog(struct bpf_verifier_log *log, diff --git a/include/linux/filter.h b/include/linux/filter.h index 259377723603..cdbc2ca8db4f 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -534,7 +534,8 @@ struct bpf_prog { is_func:1, /* program is a bpf function */ kprobe_override:1, /* Do we override a kprobe? */ has_callchain_buf:1, /* callchain buffer allocated? */ - enforce_expected_attach_type:1; /* Enforce expected_attach_type checking at attach time */ + enforce_expected_attach_type:1, /* Enforce expected_attach_type checking at attach time */ + redirect_tail_call:1; enum bpf_prog_type type; /* Type of BPF program */ enum bpf_attach_type expected_attach_type; /* For some prog types */ u32 len; /* Number of filter blocks */ @@ -613,6 +614,9 @@ struct bpf_redirect_info { void *tgt_value; struct bpf_map *map; u32 kern_flags; + struct bpf_prog *xdp_prog_redirect_tailcall; // if !NULL, we can do "everything" from BPF context. + struct xdp_buff *xdp; + struct net_device *dev; }; DECLARE_PER_CPU(struct bpf_redirect_info, bpf_redirect_info); @@ -620,6 +624,19 @@ DECLARE_PER_CPU(struct bpf_redirect_info, bpf_redirect_info); /* flags for bpf_redirect_info kern_flags */ #define BPF_RI_F_RF_NO_DIRECT BIT(0) /* no napi_direct on return_frame */ + +static inline void xdp_set_redirect_tailcall(struct bpf_prog *xdp_prog, + struct xdp_buff *xdp, + struct net_device *dev) +{ + struct bpf_redirect_info *ri = this_cpu_ptr(&bpf_redirect_info); + + ri->xdp_prog_redirect_tailcall = xdp_prog; + ri->xdp = xdp; + ri->dev = dev; +} + + /* Compute the linear packet data range [data, data_end) which * will be accessed by various program types (cls_bpf, act_bpf, * lwt, ...). Subsystems allowing direct data access must (!) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 5c7bbaac81ef..27a4fd5c8b8b 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -4594,6 +4594,54 @@ static int check_reference_leak(struct bpf_verifier_env *env) return state->acquired_refs ? -EINVAL : 0; } +static void check_redirect_map_tail_call(struct bpf_verifier_env *env, int func_id, + int insn_idx) +{ + struct bpf_insn *insns = env->prog->insnsi; + int insn_cnt = env->prog->len; + bool is_tail_call = false; + struct bpf_insn *insn; + + if (func_id != BPF_FUNC_redirect_map) + return; + + /* Naive peephole tail call checking */ + insn_idx++; +again: + if (insn_idx >= insn_cnt) + goto out; + + insn = &insns[insn_idx]; + insn_idx++; + + switch (insn->code) { + /* 1. Is the instruction following the call, an exit? */ + case BPF_JMP | BPF_EXIT: + is_tail_call = true; + break; + /* 2. True branch of "if w0 != 0 goto pc+X", an exit? */ + case BPF_JMP | BPF_JSGT | BPF_K: + case BPF_JMP32 | BPF_JSGT | BPF_K: + if (insn->dst_reg == BPF_REG_0 && insn->imm == 0) { + insn_idx += insn->off; + goto again; + } + break; + /* 3... more checks. XXX */ + default: + break; + } + +out: + if (!env->redirect_map_call_cnt++) { + env->all_redirect_map_tail = is_tail_call; + return; + } + + if (!is_tail_call) + env->all_redirect_map_tail = false; +} + static int check_helper_call(struct bpf_verifier_env *env, int func_id, int insn_idx) { const struct bpf_func_proto *fn = NULL; @@ -4685,6 +4733,8 @@ static int check_helper_call(struct bpf_verifier_env *env, int func_id, int insn } } + check_redirect_map_tail_call(env, func_id, insn_idx); + regs = cur_regs(env); /* check that flags argument in get_local_storage(map, flags) is 0, @@ -11064,6 +11114,9 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, if (ret == 0) adjust_btf_func(env); + if (ret == 0) + env->prog->redirect_tail_call = env->all_redirect_map_tail; + err_release_maps: if (!env->prog->aux->used_maps) /* if we didn't copy map pointers into bpf_prog_info, release diff --git a/net/core/filter.c b/net/core/filter.c index d01a244b5087..8cd8dee9359a 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -3701,6 +3701,60 @@ static const struct bpf_func_proto bpf_xdp_redirect_proto = { .arg2_type = ARG_ANYTHING, }; +static u64 __bpf_xdp_redirect_map_tailcall(struct bpf_map *map, u32 index, + u64 flags, + struct bpf_redirect_info *ri) +{ + struct xdp_buff *xdp = ri->xdp; + struct net_device *d = ri->dev; + struct bpf_prog *xdp_prog; + void *val; + int err; + + switch (map->map_type) { + case BPF_MAP_TYPE_DEVMAP: { + val = __dev_map_lookup_elem(map, index); + if (unlikely(!val)) + return flags; + err = dev_map_enqueue(val, xdp, d); + break; + } + case BPF_MAP_TYPE_DEVMAP_HASH: { + val = __dev_map_hash_lookup_elem(map, index); + if (unlikely(!val)) + return flags; + err = dev_map_enqueue(val, xdp, d); + break; + } + case BPF_MAP_TYPE_CPUMAP: { + val = __cpu_map_lookup_elem(map, index); + if (unlikely(!val)) + return flags; + err = cpu_map_enqueue(val, xdp, d); + break; + } + case BPF_MAP_TYPE_XSKMAP: { + val = __xsk_map_lookup_elem(map, index); + if (unlikely(!val)) + return flags; + err = __xsk_map_redirect(val, xdp); + break; + } + default: + return flags; + } + + xdp_prog = ri->xdp_prog_redirect_tailcall; + if (unlikely(err)) + goto err; + + _trace_xdp_redirect_map(d, xdp_prog, val, map, index); + return XDP_REDIRECT; +err: + _trace_xdp_redirect_map_err(d, xdp_prog, val, map, index, err); + return XDP_DROP; +} + BPF_CALL_3(bpf_xdp_redirect_map, struct bpf_map *, map, u32, ifindex, u64, flags) { @@ -3710,6 +3764,9 @@ BPF_CALL_3(bpf_xdp_redirect_map, struct bpf_map *, map, u32, ifindex, if (unlikely(flags > XDP_TX)) return XDP_ABORTED; + if (ri->xdp_prog_redirect_tailcall) + return __bpf_xdp_redirect_map_tailcall(map, ifindex, flags, ri); + ri->tgt_value = __xdp_map_lookup_elem(map, ifindex); if (unlikely(!ri->tgt_value)) { /* If the lookup fails we want to clear out the state in the From patchwork Tue Jun 9 17:26:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 1306179 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: patchwork-incoming-netdev@ozlabs.org Delivered-To: patchwork-incoming-netdev@ozlabs.org Authentication-Results: ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=vger.kernel.org (client-ip=23.128.96.18; helo=vger.kernel.org; envelope-from=netdev-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20161025 header.b=VlxiEpM4; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by ozlabs.org (Postfix) with ESMTP id 49hH9Q2vp9z9sQx for ; Wed, 10 Jun 2020 03:26:50 +1000 (AEST) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731739AbgFIR0t (ORCPT ); Tue, 9 Jun 2020 13:26:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37792 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726938AbgFIR0q (ORCPT ); Tue, 9 Jun 2020 13:26:46 -0400 Received: from mail-pg1-x543.google.com (mail-pg1-x543.google.com [IPv6:2607:f8b0:4864:20::543]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 12972C05BD1E; Tue, 9 Jun 2020 10:26:46 -0700 (PDT) Received: by mail-pg1-x543.google.com with SMTP id r18so4318898pgk.11; Tue, 09 Jun 2020 10:26:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=bL2979gbrbVOBW8YKdrznYz9+T/S2jMO+jy1PFJQ0G4=; b=VlxiEpM4wyiBp1Yv3uLk23W3Sa36G+oj8LZJMLwa+wzNOByjlQbrvDKBp9dwFa4beM qV0oLjyiBpBSHYqMsp/3megmOX7teYM8sLo/79cdIE7XlFTD3dKl5nMGY3OMyEyNm1ec baSbmlOvlQQVw4N0aP7i9vydkZAjqnOQl0jrmGt8ONNtzUbWlEpR/E55wzLzF5lJzqz2 IfARyrxHVM7dfHAtWSumauDKpJySBnr9x5hzopMnyPn4y/I3MSW4jUDpUk8UR7Shq5Xo 6omcsNAuaBqILWOtaBgM8KT5U3qcD0vjBcG0H0zuc1ucVWk7bURPlCHq7i2c0kUQ39d0 9CXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=bL2979gbrbVOBW8YKdrznYz9+T/S2jMO+jy1PFJQ0G4=; b=KAqH1hf/HIeATbry9ZHSE/DasySJXOOu6xyaRRIDDSZVyA9mRxswOqfjkflXOhmve4 II0R2SmFZcPeb0FQYMUCkTElq7CCj1GzXt9WOvsqon4EcRlkxnSBYv8uZRxXPWBIAkgw Rymqji1RJ8cGQIVmRJ3TcPzI3CyRZhqYyzHrIF5b8aSqtMKRdkaCQgT+d3IXMm6PjXhD BRk6AROONwUBr+fcql8rO0cY2QN37gB3WRVXpmEljoR999JB4PslV67hRlP5BAkbopS6 G4Jtd4ctKpNptBTL0MppUF2GLMXaj+PBbpNg63v6ZWWFyPFhThpigjbv5eafXs9bpHgP ufIQ== X-Gm-Message-State: AOAM531VWkdrXTBbqKfZNLsEG1hgOpot0TdN1Y+ObGR/LuZkKFvh/2vr IEmd+6zVgYNDiT6yjU6FUKA= X-Google-Smtp-Source: ABdhPJykY1H7+bK1nK8pg3FL2hhSWCt3PJa6hRvX7q51+XA2U0grz7LC1MW3Eq060R1hPQ2cM2CeTg== X-Received: by 2002:a63:f305:: with SMTP id l5mr24255345pgh.387.1591723605637; Tue, 09 Jun 2020 10:26:45 -0700 (PDT) Received: from btopel-mobl.ger.intel.com ([192.55.55.41]) by smtp.gmail.com with ESMTPSA id w190sm10390387pfw.35.2020.06.09.10.26.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Jun 2020 10:26:44 -0700 (PDT) From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: ast@kernel.org, daniel@iogearbox.net, bpf@vger.kernel.org, john.fastabend@gmail.com, toke@redhat.com Cc: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , magnus.karlsson@intel.com, netdev@vger.kernel.org, brouer@redhat.com, maciej.fijalkowski@intel.com Subject: [RFC PATCH bpf-next 2/2] i40e: avoid xdp_do_redirect() call when "redirect_tail_call" is set Date: Tue, 9 Jun 2020 19:26:22 +0200 Message-Id: <20200609172622.37990-3-bjorn.topel@gmail.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200609172622.37990-1-bjorn.topel@gmail.com> References: <20200609172622.37990-1-bjorn.topel@gmail.com> MIME-Version: 1.0 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org From: Björn Töpel If an XDP program, where all the bpf_redirect_map() calls are tail calls (as defined by the previous commit), the driver does not need to explicitly call xdp_do_redirect(). The driver checks the active XDP program, and notifies the BPF helper indirectly via xdp_set_redirect_tailcall(). This is just a naive, as-simple-as-possible implementation, calling xdp_set_redirect_tailcall() for each packet. For the AF_XDP rxdrop benchmark the pps went from 21.5 to 23.2 Mpps. Signed-off-by: Björn Töpel --- drivers/net/ethernet/intel/i40e/i40e_txrx.c | 14 ++++++++++++-- drivers/net/ethernet/intel/i40e/i40e_xsk.c | 14 ++++++++++++-- 2 files changed, 24 insertions(+), 4 deletions(-) diff --git a/drivers/net/ethernet/intel/i40e/i40e_txrx.c b/drivers/net/ethernet/intel/i40e/i40e_txrx.c index f9555c847f73..0d8dbdc4f47e 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_txrx.c +++ b/drivers/net/ethernet/intel/i40e/i40e_txrx.c @@ -2186,6 +2186,7 @@ static struct sk_buff *i40e_run_xdp(struct i40e_ring *rx_ring, int err, result = I40E_XDP_PASS; struct i40e_ring *xdp_ring; struct bpf_prog *xdp_prog; + bool impl_redir = false; u32 act; rcu_read_lock(); @@ -2194,6 +2195,11 @@ static struct sk_buff *i40e_run_xdp(struct i40e_ring *rx_ring, if (!xdp_prog) goto xdp_out; + if (xdp_prog->redirect_tail_call) { + impl_redir = true; + xdp_set_redirect_tailcall(rx_ring->xdp_prog, xdp, + rx_ring->netdev); + } prefetchw(xdp->data_hard_start); /* xdp_frame write */ act = bpf_prog_run_xdp(xdp_prog, xdp); @@ -2205,8 +2211,12 @@ static struct sk_buff *i40e_run_xdp(struct i40e_ring *rx_ring, result = i40e_xmit_xdp_tx_ring(xdp, xdp_ring); break; case XDP_REDIRECT: - err = xdp_do_redirect(rx_ring->netdev, xdp, xdp_prog); - result = !err ? I40E_XDP_REDIR : I40E_XDP_CONSUMED; + if (impl_redir) { + result = I40E_XDP_REDIR; + } else { + err = xdp_do_redirect(rx_ring->netdev, xdp, xdp_prog); + result = !err ? I40E_XDP_REDIR : I40E_XDP_CONSUMED; + } break; default: bpf_warn_invalid_xdp_action(act); diff --git a/drivers/net/ethernet/intel/i40e/i40e_xsk.c b/drivers/net/ethernet/intel/i40e/i40e_xsk.c index 7276580cbe64..0826f551649f 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_xsk.c +++ b/drivers/net/ethernet/intel/i40e/i40e_xsk.c @@ -146,6 +146,7 @@ static int i40e_run_xdp_zc(struct i40e_ring *rx_ring, struct xdp_buff *xdp) int err, result = I40E_XDP_PASS; struct i40e_ring *xdp_ring; struct bpf_prog *xdp_prog; + bool impl_redir = false; u32 act; rcu_read_lock(); @@ -153,6 +154,11 @@ static int i40e_run_xdp_zc(struct i40e_ring *rx_ring, struct xdp_buff *xdp) * this path is enabled by setting an XDP program. */ xdp_prog = READ_ONCE(rx_ring->xdp_prog); + if (xdp_prog->redirect_tail_call) { + impl_redir = true; + xdp_set_redirect_tailcall(rx_ring->xdp_prog, xdp, + rx_ring->netdev); + } act = bpf_prog_run_xdp(xdp_prog, xdp); switch (act) { @@ -163,8 +169,12 @@ static int i40e_run_xdp_zc(struct i40e_ring *rx_ring, struct xdp_buff *xdp) result = i40e_xmit_xdp_tx_ring(xdp, xdp_ring); break; case XDP_REDIRECT: - err = xdp_do_redirect(rx_ring->netdev, xdp, xdp_prog); - result = !err ? I40E_XDP_REDIR : I40E_XDP_CONSUMED; + if (impl_redir) { + result = I40E_XDP_REDIR; + } else { + err = xdp_do_redirect(rx_ring->netdev, xdp, xdp_prog); + result = !err ? I40E_XDP_REDIR : I40E_XDP_CONSUMED; + } break; default: bpf_warn_invalid_xdp_action(act);