From patchwork Fri Mar 30 03:15:20 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Leo Yan X-Patchwork-Id: 893097 Return-Path: X-Original-To: incoming-imx@patchwork.ozlabs.org Delivered-To: patchwork-incoming-imx@bilbo.ozlabs.org Authentication-Results: ozlabs.org; spf=none (mailfrom) smtp.mailfrom=lists.infradead.org (client-ip=2607:7c80:54:e::133; helo=bombadil.infradead.org; envelope-from=linux-arm-kernel-bounces+incoming-imx=patchwork.ozlabs.org@lists.infradead.org; receiver=) Authentication-Results: ozlabs.org; dmarc=fail (p=none dis=none) header.from=linaro.org Authentication-Results: ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="XC5REL9m"; dkim=fail reason="signature verification failed" (1024-bit key; unprotected) header.d=linaro.org header.i=@linaro.org header.b="jBkzzMQB"; dkim-atps=neutral Received: from bombadil.infradead.org (bombadil.infradead.org [IPv6:2607:7c80:54:e::133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ozlabs.org (Postfix) with ESMTPS id 40C6Gp5cvvz9s16 for ; Fri, 30 Mar 2018 14:16:50 +1100 (AEDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=7zojxfENfStgUBEM9wEblkiZ2XuQsehISK24ma5SrQQ=; b=XC5REL9mrtm8NZ pVdkku4qhZASXOOZiFR2q9jDzo2hxiK4OcX9bfmhuGhBAB7EdzvMbpq+cfwcIZ1iilnJKQaoXUG+U tYtTN/iXjzRPps9Gn1JWLLyq8npyGc7PqEheLiY/cMwvhZGt70cS1Xb0xYsElGg4PQGbOIHKngV/b 05Sh3E55PPllHEv5JQ41HDx3ROqlncC5aA9kpSE9Jwk57yJJ6auXZ9mealCHzENSWWuFaLkS4GPP0 cl3uAjDxkBOzHP9KXmzY0uJ6N/nZct28c85iYyVw+tbmPOj0TWxTi5AsJnlDlAJSeUH8T8Ddq+ZAv TiWmSkli8Qqqvsxs6mLQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1f1kWw-0000DU-Cz; Fri, 30 Mar 2018 03:16:42 +0000 Received: from mail-wm0-x243.google.com ([2a00:1450:400c:c09::243]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1f1kWN-0008Le-Sm for linux-arm-kernel@lists.infradead.org; Fri, 30 Mar 2018 03:16:10 +0000 Received: by mail-wm0-x243.google.com with SMTP id p9so13677223wmc.3 for ; Thu, 29 Mar 2018 20:15:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=MqNmyguGfYhu4/E+EGeRniZSAKHdV2m4kZR7MbrMg1Y=; b=jBkzzMQBz6gwW/Wu4VOObgFowCII/WUztxZxrgbZMLTOxqJMN1U37UAVFjiy8fNJqX yGKZSNgh2YWsjhHD1/52imESc5pY1P7ik5DSM2dEdQicVWx4HH/UQvtF3Hz3zPRSctZC O8FHr1E4P0Z7jP1+ct6A7QxnrRK2waQVmdsNI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=MqNmyguGfYhu4/E+EGeRniZSAKHdV2m4kZR7MbrMg1Y=; b=JUJVRQlNqiWm8f+2LXl8pgnNTO/MwYjU281PXVnGhpendT+mDDKoZEWiNi52Ar8fA/ +2E02oVJ4yqqltnBgivjY8HzusgxE/BmPDzIKqZPArJNfpnwsahT/wMIoDWPyqU8SHdG keRW1GUmvop0b0kiuOUoC98Ush2Bdxrwf+lQ6LW3eBN3C2b/R3Mao5Bzolwj7zR2Vy8l QsuaDsCiBZ26Vly508Kmw97gpJOvmXY9UNd1kjKWi1JG/kwHoGo3+H4kT8heXXyzmtV4 8KmsBgYt1jQzH2Z2N0N0I0wyw6jVDy2eo+noGCgDNmRsBnZIcsULrQPOi3husssyAJ7Y /zzw== X-Gm-Message-State: AElRT7GowZb/rWXQt2/U5EaXncs43i593UVOPif0fkXALUWIdis9ZiF5 uj/mFSU/+qtbfk62f8UZWuyIzg== X-Google-Smtp-Source: AIpwx4+RC2b78jDb5hc0N6RgrpOrIvU+X3ug1g3bbKSRcqdH0spgsJWspsCWlVCEXN8sdAmwsuJmlg== X-Received: by 10.28.202.25 with SMTP id a25mr939986wmg.45.1522379756161; Thu, 29 Mar 2018 20:15:56 -0700 (PDT) Received: from localhost.localdomain (li622-172.members.linode.com. [212.71.249.172]) by smtp.gmail.com with ESMTPSA id z9sm12798903wrz.4.2018.03.29.20.15.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 29 Mar 2018 20:15:55 -0700 (PDT) From: Leo Yan To: Jonathan Corbet , Mathieu Poirier , Greg Kroah-Hartman , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, coresight@lists.linaro.org, Kim Phillips , Mike Leach Subject: [PATCH v4 2/6] doc: Add documentation for Coresight panic kdump Date: Fri, 30 Mar 2018 11:15:20 +0800 Message-Id: <1522379724-30648-3-git-send-email-leo.yan@linaro.org> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1522379724-30648-1-git-send-email-leo.yan@linaro.org> References: <1522379724-30648-1-git-send-email-leo.yan@linaro.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180329_201607_971676_511996B5 X-CRM114-Status: GOOD ( 22.73 ) X-Spam-Score: -0.1 (/) X-Spam-Report: SpamAssassin version 3.4.1 on bombadil.infradead.org summary: Content analysis details: (-0.1 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, no trust [2a00:1450:400c:c09:0:0:0:243 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Leo Yan Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+incoming-imx=patchwork.ozlabs.org@lists.infradead.org List-Id: linux-imx-kernel.lists.patchwork.ozlabs.org Add detailed documentation for Coresight panic kdump, which contains the idea for why need Coresight panic kdump and introduce the implementation of Coresight panic kdump framework; the last section is to explain what's usage. Credits to Mathieu Poirier for many suggestions since the first version patch reviewing. The suggestions include using an array to manage dump related info, this makes code scalable for more CPUs; the Coresight kdump driver and integration kdump flow with other Coresight devices also have many ideas from Mathieu. Suggested-by: Mathieu Poirier Signed-off-by: Leo Yan --- .../trace/coresight/coresight-panic-kdump.txt | 130 +++++++++++++++++++++ MAINTAINERS | 1 + 2 files changed, 131 insertions(+) create mode 100644 Documentation/trace/coresight/coresight-panic-kdump.txt diff --git a/Documentation/trace/coresight/coresight-panic-kdump.txt b/Documentation/trace/coresight/coresight-panic-kdump.txt new file mode 100644 index 0000000..c02e520 --- /dev/null +++ b/Documentation/trace/coresight/coresight-panic-kdump.txt @@ -0,0 +1,130 @@ + Coresight Panic Kdump + ===================== + + Author: Leo Yan + Date: March 29th, 2018 + +Introduction +------------ + +Coresight has different sinks for trace data, the trace data is quite useful +for postmortem debugging. Embedded Trace Buffer (ETB) is one type sink which +provides on-chip storage of trace data, usually uses SRAM as the buffer with +several KBs size; if the SoC designs to support 'Local ETF' (ARM DDI 0461B, +chapter 1.2.7), every CPU has one local ETB buffer so the per CPU trace data +can avoid being overwritten by each other. Trace Memory Controller (TMC) is +another kind sink designed as a successor to the CoreSight ETB to capture trace +into DRAM. + +After Linux kernel panic has occurred, the trace data keeps the last execution +flow before issues happen. We could consider the trace data is quite useful for +postmortem debugging, especially when we can save trace data into DRAM and rely on +kdump to preserve them into vmcore file; at the end, we can retrieve trace data +from vmcore file and "offline" to analyze the execution flow. + + +Implementation +-------------- + +Coresight panic kdump is a simple framework to support Coresight dump +functionality when panic happens, it maintains an array for the dump, every array +item is dedicated to one specific CPU by using CPU number as an index. For +'offline' recovery and analysis Coresight tracing data, except should to recovery +tracing data for sinks, we also need to know CPU tracer configurations; for this +purpose, the array item is a structure which combines source and sink device +handlers, the device handler points to Coresight device structure which contains +dump info: dump buffer base address and buffer size. Below diagram is to +present data structures relationship: + + array: coresight_kdump_nodes + +------+------+----------------------+ + | CPU0 | CPU1 | ... | + +------+------+----------------------+ + | + V + coresight_kdump_node coresight_device + +-------------------+ +-------------------+ + | source_csdev | ----------> | kdump_buf | + +-------------------+ / +-------------------+ + | sink_csdev | ----' | kdump_buf_sz | + +-------------------+ +-------------------+ + +Every CPU has its own tracer, we need save tracer registers for tracer ID and +configuration related information as metadata, the metadata is used by tracing +decoder. But the tracer has the different configuration at the different phase, +below diagram explains tracer configurations for different time points: at the +system boot phase, the tracer is disabled so its registers have not been set; +after tracer has been enabled or when panic happens, tracer registers have been +configured, but we need to consider if there has CPU is locked up at panic phase +then this dead CPU has no chance to handle inter-processor interrupt for panic +dump; thus we choose tracer enabling phase to save tracer metadata. Coresight +panic kdump provides API coresight_kdump_source() as helper function for +recording tracer metadata. + + Boot Enabling Panic + + Timeline: ------->|----------->|----------->|-----------> + + Tracer: Disabled Configured Configured + Sink: Disabled Enabled Enabled with tracing data + | | + | `--> Tracing data dump + | + `--> Tracer metadata dump + +After enabling Coresight sink device, function coresight_kdump_sink() is used to +set sink device handler for related CPU; sink device handler points to Coresight +device structure, furthermore we can retrieve its ops structure and panic +callback 'panic_cb' in the ops structure. Coresight panic notifier takes panic CPU +tracing data with high priority to firstly invoke panic CPU sink callback function, +then the notifier iterates dump array and invoke callback functions one by one +for other CPUs. If one sink device is shared among CPUs, the sink panic +callback is invoked for the first traversed CPU node and other sequential CPUs +are skipped. + + +Usage +----- + +Build Linux kernel with enabling 'CONFIG_CORESIGHT_PANIC_KDUMP' configuration. + +After system booting up, we need firstly prepare dump-capture kernel, this can +refer doc [1] chapter 'Load the Dump-capture Kernel' for detailed steps. Then +we need enable the Coresight tracer, this can use either perf framework method +or sysFS interface, please refer doc [2] chapter 'How to use the tracer modules' +for detailed steps. + +When kernel panic happens, the panic kdump records trace data and launches +dump-capture kernel, we can utilize the dump-capture kernel to save kernel dump +file, this can refer doc [1] chapter 'Write Out the Dump File'. + +After get kernel dump file, we can use 'crash' tool + csdump.so extension to +extract trace data and generate 'perf.data' file: + + ./crash vmcore vmlinux + crash> extend csdump.so + crash> csdump output_dir + + We can see in the 'output_dir' there will generate out three files: + output_dir/ + ├── cstrace.bin -> trace raw data + ├── metadata.bin -> meta data + └── perf.data -> 'perf' format compatible file + +Finally use 'perf' tool for offline analysis: + + ./perf script -v -F cpu,event,ip,sym,symoff -i perf.data -k vmlinux --kallsyms /proc/kallsyms + [001] instructions: ffff000008559ad0 pl011_console_write+0x90 + [001] instructions: ffff000008559230 pl011_read+0x0 + [001] instructions: ffff00000855924c pl011_read+0x1c + [001] instructions: ffff000008559ae0 pl011_console_write+0xa0 + [001] instructions: ffff000008559ad0 pl011_console_write+0x90 + [001] instructions: ffff000008559230 pl011_read+0x0 + [001] instructions: ffff00000855924c pl011_read+0x1c + [001] instructions: ffff000008559ae0 pl011_console_write+0xa0 + [001] instructions: ffff000008559ad0 pl011_console_write+0x90 + [001] instructions: ffff000008559230 pl011_read+0x0 + [001] instructions: ffff00000855924c pl011_read+0x1c + +[1] Documentation/kdump/kdump.txt +[2] Documentation/trace/coresight/coresight.txt diff --git a/MAINTAINERS b/MAINTAINERS index 7ee1fdc..cc1243b 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1333,6 +1333,7 @@ S: Maintained F: drivers/hwtracing/coresight/* F: Documentation/trace/coresight/coresight.txt F: Documentation/trace/coresight/coresight-cpu-debug.txt +F: Documentation/trace/coresight/coresight-panic-kdump.txt F: Documentation/devicetree/bindings/arm/coresight.txt F: Documentation/devicetree/bindings/arm/coresight-cpu-debug.txt F: Documentation/ABI/testing/sysfs-bus-coresight-devices-*