| Message ID | 157918768284.1458396.8128704597152008763.stgit@toke.dk |
|---|---|
| Headers | show |
| Series | xdp: Introduce bulking for non-map XDP_REDIRECT | expand |
On Thu, Jan 16, 2020 at 7:14 AM Toke Høiland-Jørgensen <toke@redhat.com> wrote: > > Since commit 96360004b862 ("xdp: Make devmap flush_list common for all map > instances"), devmap flushing is a global operation instead of tied to a > particular map. This means that with a bit of refactoring, we can finally fix > the performance delta between the bpf_redirect_map() and bpf_redirect() helper > functions, by introducing bulking for the latter as well. > > This series makes this change by moving the data structure used for the bulking > into struct net_device itself, so we can access it even when there is not > devmap. Once this is done, moving the bpf_redirect() helper to use the bulking > mechanism becomes quite trivial, and brings bpf_redirect() up to the same as > bpf_redirect_map(): > > Before: After: > 1 CPU: > bpf_redirect_map: 8.4 Mpps 8.4 Mpps (no change) > bpf_redirect: 5.0 Mpps 8.4 Mpps (+68%) > 2 CPUs: > bpf_redirect_map: 15.9 Mpps 16.1 Mpps (+1% or ~no change) > bpf_redirect: 9.5 Mpps 15.9 Mpps (+67%) > > After this patch series, the only semantics different between the two variants > of the bpf() helper (apart from the absence of a map argument, obviously) is > that the _map() variant will return an error if passed an invalid map index, > whereas the bpf_redirect() helper will succeed, but drop packets on > xdp_do_redirect(). This is because the helper has no reference to the calling > netdev, so unfortunately we can't do the ifindex lookup directly in the helper. > > Changelog: > > v3: > - Switch two more fields to avoid a list_head spanning two cache lines > - Include Jesper's tracepoint patch > - Also rename xdp_do_flush_map() > - Fix a few nits from Maciej Applied. Thanks
Since commit 96360004b862 ("xdp: Make devmap flush_list common for all map instances"), devmap flushing is a global operation instead of tied to a particular map. This means that with a bit of refactoring, we can finally fix the performance delta between the bpf_redirect_map() and bpf_redirect() helper functions, by introducing bulking for the latter as well. This series makes this change by moving the data structure used for the bulking into struct net_device itself, so we can access it even when there is not devmap. Once this is done, moving the bpf_redirect() helper to use the bulking mechanism becomes quite trivial, and brings bpf_redirect() up to the same as bpf_redirect_map(): Before: After: 1 CPU: bpf_redirect_map: 8.4 Mpps 8.4 Mpps (no change) bpf_redirect: 5.0 Mpps 8.4 Mpps (+68%) 2 CPUs: bpf_redirect_map: 15.9 Mpps 16.1 Mpps (+1% or ~no change) bpf_redirect: 9.5 Mpps 15.9 Mpps (+67%) After this patch series, the only semantics different between the two variants of the bpf() helper (apart from the absence of a map argument, obviously) is that the _map() variant will return an error if passed an invalid map index, whereas the bpf_redirect() helper will succeed, but drop packets on xdp_do_redirect(). This is because the helper has no reference to the calling netdev, so unfortunately we can't do the ifindex lookup directly in the helper. Changelog: v3: - Switch two more fields to avoid a list_head spanning two cache lines - Include Jesper's tracepoint patch - Also rename xdp_do_flush_map() - Fix a few nits from Maciej v2: - Consolidate code paths and tracepoints for map and non-map redirect variants (Björn) - Add performance data for 2-CPU test (Jesper) - Move fields to avoid shifting cache lines in struct net_device (Eric) --- Jesper Dangaard Brouer (1): devmap: Adjust tracepoint for map-less queue flush Toke Høiland-Jørgensen (2): xdp: Move devmap bulk queue into struct net_device xdp: Use bulking for non-map XDP_REDIRECT and consolidate code paths drivers/net/tun.c | 4 + drivers/net/veth.c | 2 - drivers/net/virtio_net.c | 2 - include/linux/bpf.h | 13 +++- include/linux/filter.h | 10 ++- include/linux/netdevice.h | 13 ++-- include/trace/events/xdp.h | 130 +++++++++++++++++----------------------- kernel/bpf/devmap.c | 95 +++++++++++++++-------------- net/core/dev.c | 2 + net/core/filter.c | 90 ++++++---------------------- samples/bpf/xdp_monitor_kern.c | 8 +- 11 files changed, 160 insertions(+), 209 deletions(-)