From patchwork Fri Mar 9 14:13:40 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Hajnoczi X-Patchwork-Id: 145721 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by ozlabs.org (Postfix) with ESMTPS id C7F79B6F9F for ; Sat, 10 Mar 2012 01:14:34 +1100 (EST) Received: from localhost ([::1]:52612 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1S60aW-0004A6-KY for incoming@patchwork.ozlabs.org; Fri, 09 Mar 2012 09:14:32 -0500 Received: from eggs.gnu.org ([208.118.235.92]:54918) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1S60a1-0003Rd-9K for qemu-devel@nongnu.org; Fri, 09 Mar 2012 09:14:24 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1S60Zw-0007LQ-3A for qemu-devel@nongnu.org; Fri, 09 Mar 2012 09:14:00 -0500 Received: from e06smtp11.uk.ibm.com ([195.75.94.107]:60537) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1S60Zv-0007Hc-EZ for qemu-devel@nongnu.org; Fri, 09 Mar 2012 09:13:55 -0500 Received: from /spool/local by e06smtp11.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Fri, 9 Mar 2012 14:13:50 -0000 Received: from d06nrmr1707.portsmouth.uk.ibm.com (9.149.39.225) by e06smtp11.uk.ibm.com (192.168.101.141) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Fri, 9 Mar 2012 14:13:49 -0000 Received: from d06av01.portsmouth.uk.ibm.com (d06av01.portsmouth.uk.ibm.com [9.149.37.212]) by d06nrmr1707.portsmouth.uk.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id q29EDmt92269260 for ; Fri, 9 Mar 2012 14:13:48 GMT Received: from d06av01.portsmouth.uk.ibm.com (loopback [127.0.0.1]) by d06av01.portsmouth.uk.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id q29EDlT5030957 for ; Fri, 9 Mar 2012 07:13:47 -0700 Received: from localhost (dyn-9-174-218-67.manchester-maybrook.uk.ibm.com [9.174.218.67]) by d06av01.portsmouth.uk.ibm.com (8.14.4/8.13.1/NCO v10.0 AVin) with ESMTP id q29EDlfu030952; Fri, 9 Mar 2012 07:13:47 -0700 From: Stefan Hajnoczi To: Date: Fri, 9 Mar 2012 14:13:40 +0000 Message-Id: <1331302420-29737-1-git-send-email-stefanha@linux.vnet.ibm.com> X-Mailer: git-send-email 1.7.9.1 x-cbid: 12030914-5024-0000-0000-000001EF1D31 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 195.75.94.107 Cc: Anthony Liguori , Avi Kivity , kvm@vger.kernel.org, Stefan Hajnoczi Subject: [Qemu-devel] [PATCH] kvm: add flightrecorder script X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org The kvm kernel module includes a number of trace events which can be useful when debugging system behavior. Even on production systems these trace events can be used to observe guest behavior and identify the source of problems. The kvm_flightrecorder script is a command-line wrapper for the /sys/kernel/debug/tracing interface. Kernel symbols do not need to be installed. This script captures a fixed-size buffer of KVM trace events. Recent events overwrite the oldest events when the buffer size is exceeded and it is possible to leave KVM tracing enabled for any period of time with just a fixed-size buffer. If the buffer is large enough this script is a useful tool for collecting detailed information after an issue occurs with a guest. Hence the name "flight recorder". The script can also be used in 'tail' mode to simply view KVM trace events as they occur. This is handy for development and to ensure that the guest is indeed running. Signed-off-by: Stefan Hajnoczi --- scripts/kvm/kvm_flightrecorder | 126 ++++++++++++++++++++++++++++++++++++++++ 1 files changed, 126 insertions(+), 0 deletions(-) create mode 100755 scripts/kvm/kvm_flightrecorder diff --git a/scripts/kvm/kvm_flightrecorder b/scripts/kvm/kvm_flightrecorder new file mode 100755 index 0000000..7fb1c2d --- /dev/null +++ b/scripts/kvm/kvm_flightrecorder @@ -0,0 +1,126 @@ +#!/usr/bin/env python +# +# KVM Flight Recorder - ring buffer tracing script +# +# Copyright (C) 2012 IBM Corp +# +# Author: Stefan Hajnoczi +# +# This script provides a command-line interface to kvm ftrace and is designed +# to be used as a flight recorder that is always running. To start in-memory +# recording: +# +# sudo kvm_flightrecorder start 8192 # 8 MB per-cpu ring buffers +# +# The per-cpu ring buffer size can be given in KB as an optional argument to +# the 'start' subcommand. +# +# To stop the flight recorder: +# +# sudo kvm_flightrecorder stop +# +# To dump the contents of the flight recorder (this can be done when the +# recorder is stopped or while it is running): +# +# sudo kvm_flightrecorder dump >/path/to/dump.txt +# +# To observe the trace while it is running, use the 'tail' subcommand: +# +# sudo kvm_flightrecorder tail +# +# Note that the flight recorder may impact overall system performance by +# consuming CPU cycles. No disk I/O is performed since the ring buffer holds a +# fixed-size in-memory trace. + +import sys +import os + +tracing_dir = '/sys/kernel/debug/tracing' + +def trace_path(*args): + return os.path.join(tracing_dir, *args) + +def write_file(path, data): + open(path, 'wb').write(data) + +def enable_event(subsystem, event, enable): + write_file(trace_path('events', subsystem, event, 'enable'), '1' if enable else '0') + +def enable_subsystem(subsystem, enable): + write_file(trace_path('events', subsystem, 'enable'), '1' if enable else '0') + +def start_tracing(): + enable_subsystem('kvm', True) + write_file(trace_path('tracing_on'), '1') + +def stop_tracing(): + write_file(trace_path('tracing_on'), '0') + enable_subsystem('kvm', False) + write_file(trace_path('events', 'enable'), '0') + write_file(trace_path('current_tracer'), 'nop') + +def dump_trace(): + tracefile = open(trace_path('trace'), 'r') + try: + lines = True + while lines: + lines = tracefile.readlines(64 * 1024) + sys.stdout.writelines(lines) + except KeyboardInterrupt: + pass + +def tail_trace(): + try: + for line in open(trace_path('trace_pipe'), 'r'): + sys.stdout.write(line) + except KeyboardInterrupt: + pass + +def usage(): + print 'Usage: %s start [buffer_size_kb] | stop | dump | tail' % sys.argv[0] + print 'Control the KVM flight recorder tracing.' + sys.exit(0) + +def main(): + if len(sys.argv) < 2: + usage() + + cmd = sys.argv[1] + if cmd == '--version': + print 'kvm_flightrecorder version 1.0' + sys.exit(0) + + if not os.path.isdir(tracing_dir): + print 'Unable to tracing debugfs directory, try:' + print 'mount -t debugfs none /sys/kernel/debug' + sys.exit(1) + if not os.access(tracing_dir, os.W_OK): + print 'Unable to write to tracing debugfs directory, please run as root' + sys.exit(1) + + if cmd == 'start': + stop_tracing() # clean up first + + if len(sys.argv) == 3: + try: + buffer_size_kb = int(sys.argv[2]) + except ValueError: + print 'Invalid per-cpu trace buffer size in KB' + sys.exit(1) + write_file(trace_path('buffer_size_kb'), str(buffer_size_kb)) + print 'Per-CPU ring buffer size set to %d KB' % buffer_size_kb + + start_tracing() + print 'KVM flight recorder enabled' + elif cmd == 'stop': + stop_tracing() + print 'KVM flight recorder disabled' + elif cmd == 'dump': + dump_trace() + elif cmd == 'tail': + tail_trace() + else: + usage() + +if __name__ == '__main__': + sys.exit(main())