From patchwork Tue Nov 29 06:42:19 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexei Starovoitov X-Patchwork-Id: 700341 X-Patchwork-Delegate: davem@davemloft.net Return-Path: X-Original-To: patchwork-incoming@ozlabs.org Delivered-To: patchwork-incoming@ozlabs.org Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by ozlabs.org (Postfix) with ESMTP id 3tSYrQ3jWLz9tlW for ; Tue, 29 Nov 2016 17:42:30 +1100 (AEDT) Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.b="BFMn5dbd"; dkim-atps=neutral Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753066AbcK2Gm0 (ORCPT ); Tue, 29 Nov 2016 01:42:26 -0500 Received: from mail-pg0-f66.google.com ([74.125.83.66]:35459 "EHLO mail-pg0-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751317AbcK2GmY (ORCPT ); Tue, 29 Nov 2016 01:42:24 -0500 Received: by mail-pg0-f66.google.com with SMTP id p66so15421126pga.2 for ; Mon, 28 Nov 2016 22:42:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:mime-version:content-disposition :user-agent; bh=3jNrV1y5QnmzkClj5NQSLN7phZg/4J+5faQaxLNDQ6s=; b=BFMn5dbdNu+ppgdQ3WPUeAoz4n0iDW895oUCejQHJHg1ytZ/YSoFPOHFIH0G9d/vam 8SLltFAH0iiHrl0AyINx7v7xsYm1jcz+nxOyGmjGQ0WcC+sS8wQaa7ZtIDpPU1M0g20G 1LStbUQWc3llb/S0Zd/6SUGmDQdIIUCkLJ2dDyEc/++SNpZ+vz5rBeJStoKx/sXNUl8B vddJVcaA7g3AFq6EWXbfzx48MfVp6f5JhB5GdqCHdgwZapAWdMjSLVnzBcTLqSvc0VHJ 8kH+57dZFkwqTTa0YlmSm7AvG0r1zhqLEaMpRG4lhVfmDqkP4P29em1XyFyHYU1wuuwC FM9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:cc:subject:message-id:mime-version :content-disposition:user-agent; bh=3jNrV1y5QnmzkClj5NQSLN7phZg/4J+5faQaxLNDQ6s=; b=jvKuc45fGL4VxeVDRIcmj+5i7lJOmh8pAQK8RuyaKQamHwM4Qof1zLw7Xlc9FkiggH E4VAz5cAl5PCZEq1JZMxKODttufbVEM8DAKeHP3+Dji3kZVVJfZhvlqBNM93rS48s/ec 9y0mmhzf+f43cq5+rJsx6yGAbrSE4NGu2Ko3SEB5BtCXNAFD1xoDlQBwtwbPqYUxACaW lZlzvKs78XSAjs+MG6XjIbuv3NvZOcJbTQzEDduwBrR3lk/u2QcQw+wTRx9b4gONERdM VRxkkZYrUBB39i65m6/qP5qUMoSNjEBpufMO5m366kgAVF+kjALSt3xRtVIDTrBMWrT/ OVtQ== X-Gm-Message-State: AKaTC00Kp06aiZ1nKd/UClZmPwIBhMQQwrOA76BuIQ7quq6J47c2Y3ietVvpbep6RFPPYw== X-Received: by 10.84.216.80 with SMTP id f16mr57986391plj.91.1480401743547; Mon, 28 Nov 2016 22:42:23 -0800 (PST) Received: from ast-mbp.thefacebook.com ([2620:10d:c090:180::1:2cc9]) by smtp.gmail.com with ESMTPSA id w17sm73992676pgm.18.2016.11.28.22.42.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 28 Nov 2016 22:42:22 -0800 (PST) Date: Mon, 28 Nov 2016 22:42:19 -0800 From: Alexei Starovoitov To: netdev@vger.kernel.org Cc: Daniel Borkmann , Brenden Blanco , Thomas Graf , Wangnan , He Kuang , kernel-team@fb.com Subject: bpf debug info Message-ID: <20161129064217.GA20124@ast-mbp.thefacebook.com> MIME-Version: 1.0 Content-Disposition: inline User-Agent: Mutt/1.5.24 (2015-08-30) Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Hi All, The support for debug information in BPF was recently added to llvm. In order to use it recompile bpf programs with the following patch in samples/bpf/Makefile @@ -155,4 +155,4 @@ $(obj)/%.o: $(src)/%.c $(CLANG) $(NOSTDINC_FLAGS) $(LINUXINCLUDE) $(EXTRA_CFLAGS) \ -D__KERNEL__ -D__ASM_SYSREG_H -Wno-unused-value -Wno-pointer-sign \ -Wno-compare-distinct-pointer-types \ - -O2 -emit-llvm -c $< -o -| $(LLC) -march=bpf -filetype=obj -o $@ + -O2 -emit-llvm -g -c $< -o -| $(LLC) -march=bpf -filetype=obj -o $@ and compiled .o files can be consumed by standard llvm-objdump utility. $ llvm-objdump -S -no-show-raw-insn samples/bpf/xdp1_kern.o xdp1_kern.o: file format ELF64-BPF Disassembly of section xdp1: xdp_prog1: ; { 0: r2 = *(u32 *)(r1 + 4) ; void *data = (void *)(long)ctx->data; 8: r1 = *(u32 *)(r1 + 0) ; if (data + nh_off > data_end) 10: r3 = r1 18: r3 += 14 20: if r3 > r2 goto 55 ; h_proto = eth->h_proto; 28: r3 = *(u8 *)(r1 + 12) 30: r4 = *(u8 *)(r1 + 13) 38: r4 <<= 8 40: r4 |= r3 ; if (h_proto == htons(ETH_P_8021Q) || h_proto == htons(ETH_P_8021AD)) { 48: if r4 == 43144 goto 2 50: r3 = 14 58: if r4 != 129 goto 5 LBB0_3: ; if (data + nh_off > data_end) 60: r3 = r1 68: r3 += 18 70: if r3 > r2 goto 45 78: r3 = 18 ; h_proto = vhdr->h_vlan_encapsulated_proto; 80: r4 = *(u16 *)(r1 + 16) LBB0_5: 88: r5 = r4 90: r5 &= 65535 ; if (h_proto == htons(ETH_P_8021Q) || h_proto == htons(ETH_P_8021AD)) { 98: if r5 == 43144 goto 1 a0: if r5 != 129 goto 9 Notice that 'clang -S -o a.s' output and llvm-objdump disassembler were changed to use kernel verifier style, so now it should be easier to see what's going on. The main advantage of debug info is that verifier error messages are now easier to correlate to original C code. For example, say, in samples/bpf/parse_varlen.c I forgot to compare pointer into packet with data_end: If I try to run samples/bpf/test_cls_bpf.sh the verifier will complain: R0=imm0,min_value=0,max_value=0 R1=pkt(id=0,off=0,r=42) R2=pkt_end 112: (0f) r4 += r3 113: (0f) r1 += r4 114: (b7) r0 = 2 115: (69) r2 = *(u16 *)(r1 +2) invalid access to packet, off=2 size=2, R1(id=3,off=0,r=0) Now multiply 115 * 8 and convert to hex. This is address 0x398 in llvm-objdump: ; struct udphdr *udp = data + tp_off; 388: r1 += r4 390: r0 = 2 ; if (udp->dest == htons(DEFAULT_PKTGEN_UDP_PORT) || 398: r2 = *(u16 *)(r1 + 2) 3a0: if r2 == 2304 goto 16 Now it's clear which line of C code is causing the verifier to reject. It's still not obvious why register R1 is 'invalid pointer to packet'. The 'r=0' part of 'R1(id=3,off=0,r=0)' stands for zero bytes were verified to be valid in this register. Since 'if (udp + 1 > data_end)' was not done, the kernel doesn't know that there are valid bytes in the packet after 'udp' pointer. So next step is to improve verifier messages to be more human friendly. The step after is to introduce BPF_COMMENT pseudo instruction that will be ignored by the interpreter yet it will contain the text of original source code. Then llvm-objdump step won't be necessary. The bpf loader will load both instructions and pieces of C sources. Then verifier errors should be even easier to read and humans can easily understand the purpose of the program. PS A year ago He Kuang reported that dwarf emitted by bpf llvm backend is broken. Sorry it took so long to fix. It's probably still broken on big endian, since I've only tested on x86. --- a/samples/bpf/parse_varlen.c +++ b/samples/bpf/parse_varlen.c @@ -33,8 +33,8 @@ static int udp(void *data, uint64_t tp_off, void *data_end) { struct udphdr *udp = data + tp_off; - if (udp + 1 > data_end) - return 0; +// if (udp + 1 > data_end) +// return 0; if (udp->dest == htons(DEFAULT_PKTGEN_UDP_PORT) || udp->source == htons(DEFAULT_PKTGEN_UDP_PORT)) {