From patchwork Wed Nov 13 20:47:34 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 1194461 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="cfWRPvGi"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 47CxX06tDLz9sP4 for ; Thu, 14 Nov 2019 07:48:00 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726291AbfKMUsA (ORCPT ); Wed, 13 Nov 2019 15:48:00 -0500 Received: from mail-pg1-f194.google.com ([209.85.215.194]:36559 "EHLO mail-pg1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726162AbfKMUsA (ORCPT ); Wed, 13 Nov 2019 15:48:00 -0500 Received: by mail-pg1-f194.google.com with SMTP id k13so2111461pgh.3; Wed, 13 Nov 2019 12:47:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ek+TLf1CXwJrIPqivXOJg2aEotmC6xWAV555CANs9r4=; b=cfWRPvGi1tHdDVzjUMXohrkUBzD8c0z3UapFGbHilR6Eq304oTehWnhTRsZgRdK4++ OysFuJ+8iTDPullcO747oktlVddnfPeSuYlSxsiL+2nI3Dpywgl1hrNFq/PAPtimebPk 9ShAOmKJmC3H5JrEvXjWCD4RofWSr1zI8qOm1uFaw1vtWX/OmfU6SF/YXr98Dge6JO4E hTCQe7+2vDsZ0NIoJ20m5NWg/8ZkEIZw1D9CzziMjYkjjQGbZ0LZ8sz6exjAbZyYoS2M svCzaPGg1Bz8pR5y3zq29UU2fvHa2/kUyGf1WHmox3yaZuI3Z+znBNGbjodGUWhB9IcF aNpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ek+TLf1CXwJrIPqivXOJg2aEotmC6xWAV555CANs9r4=; b=mOM04EtIudb6N/F64pWluAB6OR7ZupNREdi3o7/wuoAx/l7xX9Bv6A5HezmJGh7OXV KszFaBiEPCQmsMJPj9QJIyprLd0Q7NHC4RyEDZUg3IhoT/jYPLQiZ42qa8gnbP/ltTkZ 50iVTxbAUE+ATS4DYQ8QtzSe+nanZz7fWa3pfHHKnNJ5JMpeOYiyz/ef7jLNzKNyx7sS RTWLzX/VhkjVZEOqwN5OJc4dAlEB6KPsfRGMrkdkb/sxj+iQ3YJJuha8YL/VajFJVk3Q N5SI/BcfwhNN6xRXVYtUOPtExuVBd9lZKISXEpYK/qqtzyIapvAPjBqdSbYoPA/D4q0e sV9w== X-Gm-Message-State: APjAAAU+Z17f3xanBmfCU+DOBq3yRbKummKaIREwAmZx90ItvATWIj2J TNp7WLpCY6owHQNHbqZJV9ecTtF7iho= X-Google-Smtp-Source: APXvYqxbtF0mWSC7qDYPFwRf9dTrlpGQx3rDJfihOCG+Qhj0IPQo1OxM7SdS1YLkSXwnBvJt8oqBlA== X-Received: by 2002:a17:90a:51:: with SMTP id 17mr7751200pjb.71.1573678079316; Wed, 13 Nov 2019 12:47:59 -0800 (PST) Received: from btopel-mobl.ger.intel.com ([192.55.54.40]) by smtp.gmail.com with ESMTPSA id g20sm3235861pgk.46.2019.11.13.12.47.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Nov 2019 12:47:58 -0800 (PST) From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: netdev@vger.kernel.org, ast@kernel.org, daniel@iogearbox.net Cc: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , bpf@vger.kernel.org, magnus.karlsson@gmail.com, magnus.karlsson@intel.com, jonathan.lemon@gmail.com Subject: [RFC PATCH bpf-next 1/4] bpf: teach bpf_arch_text_poke() jumps Date: Wed, 13 Nov 2019 21:47:34 +0100 Message-Id: <20191113204737.31623-2-bjorn.topel@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191113204737.31623-1-bjorn.topel@gmail.com> References: <20191113204737.31623-1-bjorn.topel@gmail.com> MIME-Version: 1.0 Sender: bpf-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org From: Björn Töpel The BPF dispatcher, introduced in future commits, hijacks a trampoline function. This commit teaches the text poker to emit jmpq in addtion to callq. Signed-off-by: Björn Töpel --- arch/x86/net/bpf_jit_comp.c | 34 +++++++++++++++++++++++++++++----- include/linux/bpf.h | 3 +++ 2 files changed, 32 insertions(+), 5 deletions(-) diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 79157d886a3e..28782a1c386e 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -481,7 +481,7 @@ static void emit_stx(u8 **pprog, u32 size, u32 dst_reg, u32 src_reg, int off) *pprog = prog; } -static int emit_call(u8 **pprog, void *func, void *ip) +static int emit_call_jmp(u8 **pprog, void *func, void *ip, u8 insn) { u8 *prog = *pprog; int cnt = 0; @@ -492,17 +492,28 @@ static int emit_call(u8 **pprog, void *func, void *ip) pr_err("Target call %p is out of range\n", func); return -EINVAL; } - EMIT1_off32(0xE8, offset); + EMIT1_off32(insn, offset); *pprog = prog; return 0; } +static int emit_call(u8 **pprog, void *func, void *ip) +{ + return emit_call_jmp(pprog, func, ip, 0xE8); +} + +/* Emits tail-call */ +static int emit_jmp(u8 **pprog, void *func, void *ip) +{ + return emit_call_jmp(pprog, func, ip, 0xE9); +} + int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t, void *old_addr, void *new_addr) { u8 old_insn[X86_CALL_SIZE] = {}; u8 new_insn[X86_CALL_SIZE] = {}; - u8 *prog; + u8 *prog, insn; int ret; if (!is_kernel_text((long)ip) && @@ -510,31 +521,44 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t, /* BPF trampoline in modules is not supported */ return -EINVAL; + switch (t) { + case BPF_MOD_NOP_TO_CALL: + case BPF_MOD_CALL_TO_CALL: + case BPF_MOD_CALL_TO_NOP: + insn = 0xE8; + break; + default: + insn = 0xE9; + } + if (old_addr) { prog = old_insn; - ret = emit_call(&prog, old_addr, (void *)ip); + ret = emit_call_jmp(&prog, old_addr, (void *)ip, insn); if (ret) return ret; } if (new_addr) { prog = new_insn; - ret = emit_call(&prog, new_addr, (void *)ip); + ret = emit_call_jmp(&prog, new_addr, (void *)ip, insn); if (ret) return ret; } ret = -EBUSY; mutex_lock(&text_mutex); switch (t) { + case BPF_MOD_NOP_TO_JMP: case BPF_MOD_NOP_TO_CALL: if (memcmp(ip, ideal_nops[NOP_ATOMIC5], X86_CALL_SIZE)) goto out; text_poke(ip, new_insn, X86_CALL_SIZE); break; + case BPF_MOD_JMP_TO_JMP: case BPF_MOD_CALL_TO_CALL: if (memcmp(ip, old_insn, X86_CALL_SIZE)) goto out; text_poke(ip, new_insn, X86_CALL_SIZE); break; + case BPF_MOD_JMP_TO_NOP: case BPF_MOD_CALL_TO_NOP: if (memcmp(ip, old_insn, X86_CALL_SIZE)) goto out; diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 6a80af092048..38b0715050a9 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1270,6 +1270,9 @@ enum bpf_text_poke_type { BPF_MOD_NOP_TO_CALL, BPF_MOD_CALL_TO_CALL, BPF_MOD_CALL_TO_NOP, + BPF_MOD_NOP_TO_JMP, + BPF_MOD_JMP_TO_JMP, + BPF_MOD_JMP_TO_NOP, }; int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t, void *addr1, void *addr2); From patchwork Wed Nov 13 20:47:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 1194463 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="Wc+F1lkU"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 47CxX727R2z9sP4 for ; Thu, 14 Nov 2019 07:48:07 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726162AbfKMUsH (ORCPT ); Wed, 13 Nov 2019 15:48:07 -0500 Received: from mail-pf1-f196.google.com ([209.85.210.196]:33715 "EHLO mail-pf1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726557AbfKMUsG (ORCPT ); Wed, 13 Nov 2019 15:48:06 -0500 Received: by mail-pf1-f196.google.com with SMTP id c184so2459971pfb.0; Wed, 13 Nov 2019 12:48:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=MZGF2jFZFxMBGCARyf2ubuGKnAZ6fZkQVyV2aX26Is8=; b=Wc+F1lkUBoJ2QEl8H2Wuum5hkE40JQZywnDmHgiNj/be8IbI4p87Adazjl7gNC0jtP 5Q2QWqHpypYLXmVa86ByOTFZCzwR1kQFK2M1s9bP0eeVOgZmrFYGqc4wUR7b0tSfGV5T 9q0v64QYTk9mt4ySeJT1SPxKEn6m46hPhB9BsdUNc9JkBAIIDyasZ14FsRFPiUcmKEn7 1jja1zAWpVyyjYXV0ZXJ8TGo6HGRgEpTDryE45M1NZ67hzauRl9ubX+X6YJlPTXlrGec 7RzNX4JGf6y28Y5w/Do7hAaIJwrRBJslDfnka/YIqz/cU39svCo2OFg5WY4zscTUumgu YriA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=MZGF2jFZFxMBGCARyf2ubuGKnAZ6fZkQVyV2aX26Is8=; b=GdCbe52CTVG9lX9ZKVOXuNafV7ik2pDGgqaoI0OJgxl6GlYlTSi9bfxvDGawlkSywB JwW/4NKQlITjiDu/c0GepBZyZlI0VCk+tmsSSbHjx0rQt0Yo/S7FznTR0wGgVPlji5xK 0eeFU+MphQquN994IPNF+fRz9Wkbzsyoyn2cubZzeUU3CEm1ARCXMhxZ65dFU5QOvfaf zCEXTBFFv7sb7fv6kkdXjUHMxYxqdO/zgMiDo0iCsRpXrYJQQAYeQlD94IRYUlZoqo5v dE0qkuJt7+N8JnsMRxmiOIZr8+VIZAMztDm3c/y4C0/mWEYCwwExDkct4uqRWU78cxbD pGmA== X-Gm-Message-State: APjAAAVNDNwlbBIcGismNLu/cP+i78nxs/qm5GaQAe8piRQ4Z+UvOiq4 UQ5tfGen3wJGWLk6RjYDj0GDb8MXsCE= X-Google-Smtp-Source: APXvYqxQKvSumY6+pys06JJCco9VmGg/1+FtdWCbw34GC58upxTgQGSxQJ1HD76yauvhXrZwoFlCgw== X-Received: by 2002:a63:535a:: with SMTP id t26mr5238810pgl.215.1573678083924; Wed, 13 Nov 2019 12:48:03 -0800 (PST) Received: from btopel-mobl.ger.intel.com ([192.55.54.40]) by smtp.gmail.com with ESMTPSA id g20sm3235861pgk.46.2019.11.13.12.47.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Nov 2019 12:48:03 -0800 (PST) From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: netdev@vger.kernel.org, ast@kernel.org, daniel@iogearbox.net Cc: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , bpf@vger.kernel.org, magnus.karlsson@gmail.com, magnus.karlsson@intel.com, jonathan.lemon@gmail.com Subject: [RFC PATCH bpf-next 2/4] bpf: introduce BPF dispatcher Date: Wed, 13 Nov 2019 21:47:35 +0100 Message-Id: <20191113204737.31623-3-bjorn.topel@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191113204737.31623-1-bjorn.topel@gmail.com> References: <20191113204737.31623-1-bjorn.topel@gmail.com> MIME-Version: 1.0 Sender: bpf-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org From: Björn Töpel The BPF dispatcher builds on top of the BPF trampoline ideas; Introduce bpf_arch_text_poke() and (re-)use the BPF JIT generate code. The dispatcher builds a dispatch table for XDP programs, for retpoline avoidance. The table is a simple binary search model, so lookup is O(log n). Here, the dispatch table is limited to four entries (for laziness reason -- only 1B relative jumps :-P). If the dispatch table is full, it will fallback to the retpoline path. An example: A module/driver allocates a dispatcher. The dispatcher is shared for all netdevs. Each netdev allocate a slot in the dispatcher and a BPF program. The netdev then uses the dispatcher to call the correct program with a direct call (actually a tail-call). Signed-off-by: Björn Töpel --- arch/x86/net/bpf_jit_comp.c | 96 ++++++++++++++++++ kernel/bpf/Makefile | 1 + kernel/bpf/dispatcher.c | 197 ++++++++++++++++++++++++++++++++++++ 3 files changed, 294 insertions(+) create mode 100644 kernel/bpf/dispatcher.c diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 28782a1c386e..d75aebf508b8 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -10,10 +10,12 @@ #include #include #include +#include #include #include #include #include +#include static u8 *emit_code(u8 *ptr, u32 bytes, unsigned int len) { @@ -1471,6 +1473,100 @@ int arch_prepare_bpf_trampoline(void *image, struct btf_func_model *m, u32 flags return 0; } +#if defined(CONFIG_BPF_JIT) && defined(CONFIG_RETPOLINE) + +/* Emits the dispatcher. Id lookup is limited to BPF_DISPATCHER_MAX, + * so it'll fit into PAGE_SIZE/2. The lookup is binary search: O(log + * n). + */ +static int emit_bpf_dispatcher(u8 **pprog, int a, int b, u64 *progs, + u8 *fb) +{ + u8 *prog = *pprog, *jg_reloc; + int pivot, err, cnt = 0; + s64 jmp_offset; + + if (a == b) { + emit_mov_imm64(&prog, BPF_REG_0, /* movabs func,%rax */ + progs[a] >> 32, + (progs[a] << 32) >> 32); + EMIT3(0x48, 0x39, 0xC2); /* cmp rdx, rax */ + jmp_offset = fb - (prog + 2); + if (!is_imm8(jmp_offset)) + return -1; + EMIT2(X86_JNE, jmp_offset); /* jne retpoline */ + err = emit_jmp(&prog, (void *)progs[a], /* jmp bpf_prog */ + prog); + if (err) + return err; + goto out; + } + + pivot = (b - a) / 2; + emit_mov_imm64(&prog, BPF_REG_0, progs[a + pivot] >> 32, + (progs[a + pivot] << 32) >> 32); + EMIT3(0x48, 0x39, 0xC2); /* cmp rdx, rax */ + + jg_reloc = prog; + EMIT2(X86_JG, 0); /* jg pivot + 1-part */ + + err = emit_bpf_dispatcher(&prog, a, a + pivot, progs, fb); + if (err) + return err; + + jmp_offset = prog - (jg_reloc + 2); + if (!is_imm8(jmp_offset)) + return -1; + emit_code(jg_reloc, X86_JG + (jmp_offset << 8), 2); + + err = emit_bpf_dispatcher(&prog, a + pivot + 1, b, progs, fb); + if (err) + return err; +out: + *pprog = prog; + return 0; +} + +#endif + +static int cmp_ips(const void *a, const void *b) +{ + const u64 *ipa = a; + const u64 *ipb = b; + + if (*ipa > *ipb) + return 1; + if (*ipa < *ipb) + return -1; + return 0; +} + +#define BPF_DISPATCHER_MAX 4 + +int arch_prepare_bpf_dispatcher(void *image, struct bpf_prog **progs, + int num_progs) +{ + u64 ips[BPF_DISPATCHER_MAX] = {}; + u8 *fallback, *prog = image; + int i, err, cnt = 0; + + if (!num_progs || num_progs > BPF_DISPATCHER_MAX) + return -EINVAL; + + for (i = 0; i < num_progs; i++) + ips[i] = (u64)progs[i]->bpf_func; + + EMIT2(0xEB, 5); /* jmp rip+5 (skip retpoline) */ + fallback = prog; + err = emit_jmp(&prog, /* jmp retpoline */ + __x86_indirect_thunk_rdx, prog); + if (err) + return err; + + sort(&ips[0], num_progs, sizeof(ips[i]), cmp_ips, NULL); + return emit_bpf_dispatcher(&prog, 0, num_progs - 1, &ips[0], fallback); +} + struct x64_jit_data { struct bpf_binary_header *header; int *addrs; diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile index 3f671bf617e8..d4f330351f87 100644 --- a/kernel/bpf/Makefile +++ b/kernel/bpf/Makefile @@ -8,6 +8,7 @@ obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o obj-$(CONFIG_BPF_SYSCALL) += disasm.o obj-$(CONFIG_BPF_JIT) += trampoline.o obj-$(CONFIG_BPF_SYSCALL) += btf.o +obj-$(CONFIG_BPF_JIT) += dispatcher.o ifeq ($(CONFIG_NET),y) obj-$(CONFIG_BPF_SYSCALL) += devmap.o obj-$(CONFIG_BPF_SYSCALL) += cpumap.o diff --git a/kernel/bpf/dispatcher.c b/kernel/bpf/dispatcher.c new file mode 100644 index 000000000000..691898640720 --- /dev/null +++ b/kernel/bpf/dispatcher.c @@ -0,0 +1,197 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2019 Intel Corporation. */ + +#ifdef CONFIG_RETPOLINE + +#include +#include +#include + +/* The BPF dispatcher is a multiway branch code generator. A user + * registers a slot (id) and can then update the BPF program for that + * slot. The dispatcher is jited, and will be rejited every time a + * slot is allocated/deallocated for performance reasons. An example: + * A module provides code for multiple netdevs. Each netdev can have + * one XDP program. The module code will allocate a dispatcher, and + * when the netdev enables XDP it allocates a new slot. + * + * Nothing like STATIC_CALL_INLINE is supported yet, so an explicit + * trampoline is needed: + * + * unsigned int dispatcher_trampoline(void *ctx, void *insn, int id) + */ + +#define DISPATCHER_HASH_BITS 10 +#define DISPATCHER_TABLE_SIZE (1 << DISPATCHER_HASH_BITS) + +static struct hlist_head dispatcher_table[DISPATCHER_TABLE_SIZE]; + +#define BPF_DISPATCHER_MAX 4 + +struct bpf_dispatcher { + struct hlist_node hlist; + void *func; + struct bpf_prog *progs[BPF_DISPATCHER_MAX]; + int num_progs; + void *image; + u64 selector; +}; + +static DEFINE_MUTEX(dispatcher_mutex); + +struct bpf_dispatcher *bpf_dispatcher_lookup(void *func) +{ + struct bpf_dispatcher *d; + struct hlist_head *head; + void *image; + + head = &dispatcher_table[hash_ptr(func, DISPATCHER_HASH_BITS)]; + hlist_for_each_entry(d, head, hlist) { + if (d->func == func) + return d; + } + d = kzalloc(sizeof(*d), GFP_KERNEL); + if (!d) + return NULL; + + image = bpf_jit_alloc_exec(PAGE_SIZE); + if (!image) { + kfree(d); + return NULL; + } + + d->func = func; + INIT_HLIST_NODE(&d->hlist); + hlist_add_head(&d->hlist, head); + + set_vm_flush_reset_perms(image); + set_memory_x((long)image, 1); + d->image = image; + return d; +} + +static void bpf_dispatcher_free(struct bpf_dispatcher *d) +{ + bpf_jit_free_exec(d->image); + hlist_del(&d->hlist); + kfree(d); +} + +static int bpf_dispatcher_add_prog(struct bpf_dispatcher *d, + struct bpf_prog *prog) +{ + struct bpf_prog **entry = NULL; + int i, err = 0; + + if (d->num_progs == BPF_DISPATCHER_MAX) + return err; + + for (i = 0; i < BPF_DISPATCHER_MAX; i++) { + if (!entry && !d->progs[i]) + entry = &d->progs[i]; + if (d->progs[i] == prog) + return err; + } + + prog = bpf_prog_inc(prog); + if (IS_ERR(prog)) + return err; + + *entry = prog; + d->num_progs++; + return err; +} + +static void bpf_dispatcher_remove_prog(struct bpf_dispatcher *d, + struct bpf_prog *prog) +{ + int i; + + for (i = 0; i < BPF_DISPATCHER_MAX; i++) { + if (d->progs[i] == prog) { + bpf_prog_put(prog); + d->progs[i] = NULL; + d->num_progs--; + break; + } + } +} + +int __weak arch_prepare_bpf_dispatcher(void *image, struct bpf_prog **progs, + int num_ids) +{ + return -ENOTSUPP; +} + +/* NB! bpf_dispatcher_update() might free the dispatcher! */ +static int bpf_dispatcher_update(struct bpf_dispatcher *d) +{ + void *old_image = d->image + ((d->selector + 1) & 1) * PAGE_SIZE / 2; + void *new_image = d->image + (d->selector & 1) * PAGE_SIZE / 2; + int err; + + if (d->num_progs == 0) { + err = bpf_arch_text_poke(d->func, BPF_MOD_JMP_TO_NOP, + old_image, NULL); + bpf_dispatcher_free(d); + goto out; + } + + err = arch_prepare_bpf_dispatcher(new_image, &d->progs[0], + d->num_progs); + if (err) + goto out; + + if (d->selector) + /* progs already running at this address */ + err = bpf_arch_text_poke(d->func, BPF_MOD_JMP_TO_JMP, + old_image, new_image); + else + /* first time registering */ + err = bpf_arch_text_poke(d->func, BPF_MOD_NOP_TO_JMP, + NULL, new_image); + + if (err) + goto out; + d->selector++; + +out: + return err; +} + +void bpf_dispatcher_change_prog(void *func, struct bpf_prog *from, + struct bpf_prog *to) +{ + struct bpf_dispatcher *d; + + if (!from && !to) + return; + + mutex_lock(&dispatcher_mutex); + d = bpf_dispatcher_lookup(func); + if (!d) + goto out; + + if (from) + bpf_dispatcher_remove_prog(d, from); + + if (to) + bpf_dispatcher_add_prog(d, to); + + WARN_ON(bpf_dispatcher_update(d)); + +out: + mutex_unlock(&dispatcher_mutex); +} + +static int __init init_dispatchers(void) +{ + int i; + + for (i = 0; i < DISPATCHER_TABLE_SIZE; i++) + INIT_HLIST_HEAD(&dispatcher_table[i]); + return 0; +} +late_initcall(init_dispatchers); + +#endif From patchwork Wed Nov 13 20:47:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 1194465 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="h0ZELYWt"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 47CxXC19VSz9sP4 for ; Thu, 14 Nov 2019 07:48:11 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726755AbfKMUsK (ORCPT ); Wed, 13 Nov 2019 15:48:10 -0500 Received: from mail-pf1-f193.google.com ([209.85.210.193]:39933 "EHLO mail-pf1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726613AbfKMUsK (ORCPT ); Wed, 13 Nov 2019 15:48:10 -0500 Received: by mail-pf1-f193.google.com with SMTP id x28so2443805pfo.6; Wed, 13 Nov 2019 12:48:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=RDqExjLSfa62qPvgMCMSCU7ZsksGthSKOKxyyHhHC1g=; b=h0ZELYWtAaPORy0Z8QXqOtjTwfEid9xamAbfY0h99lpF703hzanlWGCcwRaOKKPIDx 6+TgbgdVGWdne9EdwtnVyii84ECPlZb08MXwY0WctH+2fT46+zA1yi8BTKR12oSwnMeY OT8W3EfOJ8Dy1IpX9W8hvVtAKzcQ3mNdZL6D2hoRvcucYBWM5q0JfxhGHgRFehDhL43n 83xiNjVAC0t/c8LpJKKFqmxg8FoIPBb6VTEbYGarmPGvAvplLxBjcg86l+XRr1wZQyal Uy8tVQqXXBPAJZ2ZRZSdMHoJb2J4AnUTfKWUIGnqGRVbMvRLp4I0X26wB1DOib3HQBEZ KMfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=RDqExjLSfa62qPvgMCMSCU7ZsksGthSKOKxyyHhHC1g=; b=BKFaL8O2rD3N9fYD4lGjrmY0mX6ap1r8vrmxAt6Ouokt6NUjr7x644F1sJsH/fA7Km DWu2QFJMuImDnvcha/sbKqk0TFV34T1slYcQyR21MZC9cH8T4juSi4v0C4dRLC+E3Boo blyXM/nK3EgPQoktItxj6DqQJ9N9PVsG/KkjWkhXMn6b+lZdfH7U4jYvl5zf1x965u3o lHHP6a9cDCHw5PHeJMKZtk7q1xACIRa7XRWIZ6qMq+TuCVu3mWCzYjsDWdbKDxxanzpY Zf+cNirop4T+lGwxikPIf1Y0CwellDkuTDrtMh7J0C3VZfFq0li7/bbrvKplqjACSfgE kVqA== X-Gm-Message-State: APjAAAVHA0HwKTsyIsN/MM33q0i9lFmC5N9iwilLPoRoZAKUbXeg/4Kw kEUz6IfxGRe7IrHl3PfITTurS4Dz4qE= X-Google-Smtp-Source: APXvYqyjOEo10AadQlUUJTqPlEV605cRIZlVXbSPOYcf7uj1iq3BVOCI6GkLoA+PFFo/mh/qdjebrg== X-Received: by 2002:a62:8013:: with SMTP id j19mr5136727pfd.220.1573678088018; Wed, 13 Nov 2019 12:48:08 -0800 (PST) Received: from btopel-mobl.ger.intel.com ([192.55.54.40]) by smtp.gmail.com with ESMTPSA id g20sm3235861pgk.46.2019.11.13.12.48.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Nov 2019 12:48:07 -0800 (PST) From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: netdev@vger.kernel.org, ast@kernel.org, daniel@iogearbox.net Cc: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , bpf@vger.kernel.org, magnus.karlsson@gmail.com, magnus.karlsson@intel.com, jonathan.lemon@gmail.com Subject: [RFC PATCH bpf-next 3/4] xdp: introduce xdp_call Date: Wed, 13 Nov 2019 21:47:36 +0100 Message-Id: <20191113204737.31623-4-bjorn.topel@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191113204737.31623-1-bjorn.topel@gmail.com> References: <20191113204737.31623-1-bjorn.topel@gmail.com> MIME-Version: 1.0 Sender: bpf-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org From: Björn Töpel The xdp_call.h header wraps a more user-friendly API around the BPF dispatcher. A user adds a trampoline/XDP caller using the DEFINE_XDP_CALL macro, and updates the BPF dispatcher via xdp_call_update(). The actual dispatch is done via xdp_call(). The next commit will show-case how the i40e uses xdp_call. When static_call, especially the inlined version, is upstreamed it will be used by xdp_call.h. Signed-off-by: Björn Töpel --- include/linux/xdp_call.h | 49 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 include/linux/xdp_call.h diff --git a/include/linux/xdp_call.h b/include/linux/xdp_call.h new file mode 100644 index 000000000000..e736a4d3c961 --- /dev/null +++ b/include/linux/xdp_call.h @@ -0,0 +1,49 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright(c) 2019 Intel Corporation. */ +#ifndef _LINUX_XDP_CALL_H +#define _LINUX_XDP_CALL_H + +#include + +#if defined(CONFIG_BPF_JIT) && defined(CONFIG_RETPOLINE) + +void bpf_dispatcher_change_prog(void *func, struct bpf_prog *from, + struct bpf_prog *to); + +#define XDP_CALL_TRAMP(name) ____xdp_call_##name##_tramp + +#define DEFINE_XDP_CALL(name) \ + unsigned int XDP_CALL_TRAMP(name)( \ + const void *xdp_ctx, \ + const struct bpf_insn *insnsi, \ + unsigned int (*bpf_func)(const void *, \ + const struct bpf_insn *)) \ + { \ + return bpf_func(xdp_ctx, insnsi); \ + } + +#define DECLARE_XDP_CALL(name) \ + unsigned int XDP_CALL_TRAMP(name)( \ + const void *xdp_ctx, \ + const struct bpf_insn *insnsi, \ + unsigned int (*bpf_func)(const void *, \ + const struct bpf_insn *)) + +#define xdp_call_run(name, xdp_prog, xdp) \ + XDP_CALL_TRAMP(name)(xdp, xdp_prog->insnsi, \ + xdp_prog->bpf_func) + +#define xdp_call_update(name, from_xdp_prog, to_xdp_prog) \ + bpf_dispatcher_change_prog(&XDP_CALL_TRAMP(name), \ + from_xdp_prog, \ + to_xdp_prog) + +#else /* !defined(CONFIG_BPF_JIT) || !defined(CONFIG_RETPOLINE) */ + +#define DEFINE_XDP_CALL(name) +#define DECLARE_XDP_CALL(name) +#define xdp_call_run(name, xdp_prog, xdp) bpf_prog_run_xdp(xdp_prog, xdp) +#define xdp_call_update(name, from_xdp_prog, to_xdp_prog) + +#endif /* defined(CONFIG_BPF_JIT) && defined(CONFIG_RETPOLINE) */ +#endif /* _LINUX_XDP_CALL_H */ From patchwork Wed Nov 13 20:47:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 1194467 X-Patchwork-Delegate: bpf@iogearbox.net Return-Path: X-Original-To: incoming-bpf@patchwork.ozlabs.org Delivered-To: patchwork-incoming-bpf@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (no SPF record) smtp.mailfrom=vger.kernel.org (client-ip=209.132.180.67; helo=vger.kernel.org; envelope-from=bpf-owner@vger.kernel.org; receiver=) Authentication-Results: ozlabs.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="jbAxRhKP"; dkim-atps=neutral Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 47CxXF3yRsz9sPF for ; Thu, 14 Nov 2019 07:48:13 +1100 (AEDT) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726960AbfKMUsM (ORCPT ); Wed, 13 Nov 2019 15:48:12 -0500 Received: from mail-pg1-f193.google.com ([209.85.215.193]:36577 "EHLO mail-pg1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726613AbfKMUsM (ORCPT ); Wed, 13 Nov 2019 15:48:12 -0500 Received: by mail-pg1-f193.google.com with SMTP id k13so2111777pgh.3; Wed, 13 Nov 2019 12:48:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+hHPk4MDXZMe1Dt2BSBD6rDj4LAJJcjXpcIlnBfgPIE=; b=jbAxRhKPD+dzE/RXQAwYDAmeb56ZOCrSALBy/nONC1+lxemUmySj48nLZ4PcZeYC7A N4mUHIwCjRM5Im39WFsCwEVQkNelUvnbtPig8iU5UxkLS+16LN6O/7cPsEtokTJBYtVw dywEvFTD+K9WIMZfbtsztNqtLu0l6WtFw6OhpGq8Q1YkM6eMekdau/x3r92DVMNZcsnu tTrrVi7H5GYwWKs9njFjjEWt/0acZKSmbXRn+iUW5pvcKvCMtp7QkDuyHop/GAYqCnaC gw0pItTXRPx3mw1eosmBH4ps/+BdDMUDGeEWt7PbnI7VIHwe/PrdYQuH6PN72+xYdvo0 DMUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+hHPk4MDXZMe1Dt2BSBD6rDj4LAJJcjXpcIlnBfgPIE=; b=BfpOy2F/RvpLbonl46P5ZSDgUrRXVUVIfLAS5HF6RgnBZDaM/izJnIKB6Lt4DijByA ecUPT8jtvaSAuK3DW/tICC5cG1E+CC/KxxqQ8iHDuSF+xhHnSmL9uZJaarMEH58uuJIs ZWmRsh2v4Tpl6rwTpsZTFVtSJulTNtCodwP0aw2GQOgWurz++YW5nvLLlhWANF/GU9V3 RLOFyGMvMWf/fuDKYmk+dTIGuNw9fV7SjImVPrAvElTwlblTTDjB/YQoGCb8001z8HGx uelrmk9SY8nDVJENdmsBYqwQbzYOSHTjDFMMHHQpkmIipuUJPWZoAGSa4LdRAC6XDP0s eupA== X-Gm-Message-State: APjAAAXuNb7QqIJfuX4PFRZu+awDVfZbSt5NVnZOD/IX7lFBaX2H/yCF juuDErIwEIHIAI84baPpyIX+JirbwzY= X-Google-Smtp-Source: APXvYqw5ujzcd0PI2MvIwrUQlQSkj2RwmPOvvDtbyHAmP6YBtoyU8rAIngIdtlqLpRPMVz/YxnqzEQ== X-Received: by 2002:a63:8f5e:: with SMTP id r30mr6060014pgn.146.1573678091285; Wed, 13 Nov 2019 12:48:11 -0800 (PST) Received: from btopel-mobl.ger.intel.com ([192.55.54.40]) by smtp.gmail.com with ESMTPSA id g20sm3235861pgk.46.2019.11.13.12.48.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Nov 2019 12:48:10 -0800 (PST) From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: netdev@vger.kernel.org, ast@kernel.org, daniel@iogearbox.net Cc: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , bpf@vger.kernel.org, magnus.karlsson@gmail.com, magnus.karlsson@intel.com, jonathan.lemon@gmail.com Subject: [RFC PATCH bpf-next 4/4] i40e: start using xdp_call.h Date: Wed, 13 Nov 2019 21:47:37 +0100 Message-Id: <20191113204737.31623-5-bjorn.topel@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20191113204737.31623-1-bjorn.topel@gmail.com> References: <20191113204737.31623-1-bjorn.topel@gmail.com> MIME-Version: 1.0 Sender: bpf-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: bpf@vger.kernel.org From: Björn Töpel This commit starts using xdp_call.h and the BPF dispatcher to avoid the retpoline overhead. Signed-off-by: Björn Töpel --- drivers/net/ethernet/intel/i40e/i40e_main.c | 5 +++++ drivers/net/ethernet/intel/i40e/i40e_txrx.c | 5 ++++- drivers/net/ethernet/intel/i40e/i40e_xsk.c | 5 ++++- 3 files changed, 13 insertions(+), 2 deletions(-) diff --git a/drivers/net/ethernet/intel/i40e/i40e_main.c b/drivers/net/ethernet/intel/i40e/i40e_main.c index b3d7edbb1389..59b530e4198f 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_main.c +++ b/drivers/net/ethernet/intel/i40e/i40e_main.c @@ -5,6 +5,7 @@ #include #include #include +#include /* Local includes */ #include "i40e.h" @@ -12517,6 +12518,8 @@ static netdev_features_t i40e_features_check(struct sk_buff *skb, return features & ~(NETIF_F_CSUM_MASK | NETIF_F_GSO_MASK); } +DEFINE_XDP_CALL(i40e_xdp_call); + /** * i40e_xdp_setup - add/remove an XDP program * @vsi: VSI to changed @@ -12552,6 +12555,8 @@ static int i40e_xdp_setup(struct i40e_vsi *vsi, for (i = 0; i < vsi->num_queue_pairs; i++) WRITE_ONCE(vsi->rx_rings[i]->xdp_prog, vsi->xdp_prog); + xdp_call_update(i40e_xdp_call, old_prog, prog); + if (old_prog) bpf_prog_put(old_prog); diff --git a/drivers/net/ethernet/intel/i40e/i40e_txrx.c b/drivers/net/ethernet/intel/i40e/i40e_txrx.c index b8496037ef7f..34d7b15897a1 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_txrx.c +++ b/drivers/net/ethernet/intel/i40e/i40e_txrx.c @@ -3,6 +3,7 @@ #include #include +#include #include #include "i40e.h" #include "i40e_trace.h" @@ -2188,6 +2189,8 @@ int i40e_xmit_xdp_tx_ring(struct xdp_buff *xdp, struct i40e_ring *xdp_ring) return i40e_xmit_xdp_ring(xdpf, xdp_ring); } +DECLARE_XDP_CALL(i40e_xdp_call); + /** * i40e_run_xdp - run an XDP program * @rx_ring: Rx ring being processed @@ -2209,7 +2212,7 @@ static struct sk_buff *i40e_run_xdp(struct i40e_ring *rx_ring, prefetchw(xdp->data_hard_start); /* xdp_frame write */ - act = bpf_prog_run_xdp(xdp_prog, xdp); + act = xdp_call_run(i40e_xdp_call, xdp_prog, xdp); switch (act) { case XDP_PASS: break; diff --git a/drivers/net/ethernet/intel/i40e/i40e_xsk.c b/drivers/net/ethernet/intel/i40e/i40e_xsk.c index a05dfecdd9b4..c623eeaeb625 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_xsk.c +++ b/drivers/net/ethernet/intel/i40e/i40e_xsk.c @@ -2,6 +2,7 @@ /* Copyright(c) 2018 Intel Corporation. */ #include +#include #include #include @@ -179,6 +180,8 @@ int i40e_xsk_umem_setup(struct i40e_vsi *vsi, struct xdp_umem *umem, i40e_xsk_umem_disable(vsi, qid); } +DECLARE_XDP_CALL(i40e_xdp_call); + /** * i40e_run_xdp_zc - Executes an XDP program on an xdp_buff * @rx_ring: Rx ring @@ -202,7 +205,7 @@ static int i40e_run_xdp_zc(struct i40e_ring *rx_ring, struct xdp_buff *xdp) * this path is enabled by setting an XDP program. */ xdp_prog = READ_ONCE(rx_ring->xdp_prog); - act = bpf_prog_run_xdp(xdp_prog, xdp); + act = xdp_call_run(i40e_xdp_call, xdp_prog, xdp); offset = xdp->data - xdp->data_hard_start; xdp->handle = xsk_umem_adjust_offset(umem, xdp->handle, offset);