From patchwork Fri Apr 13 01:32:36 2012 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 152244 Return-Path: X-Original-To: incoming@patchwork.ozlabs.org Delivered-To: patchwork-incoming@bilbo.ozlabs.org Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (Client did not present a certificate) by ozlabs.org (Postfix) with ESMTPS id E6CE5B6FFE for ; Fri, 13 Apr 2012 12:01:04 +1000 (EST) Received: from localhost ([::1]:57807 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1SIVSJ-0000NA-80 for incoming@patchwork.ozlabs.org; Thu, 12 Apr 2012 21:37:43 -0400 Received: from eggs.gnu.org ([208.118.235.92]:39065) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1SIVRW-0006by-2R for qemu-devel@nongnu.org; Thu, 12 Apr 2012 21:36:55 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1SIVRT-0002Ey-Ux for qemu-devel@nongnu.org; Thu, 12 Apr 2012 21:36:53 -0400 Received: from mx1.redhat.com ([209.132.183.28]:18866) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1SIVRT-0002ER-MQ for qemu-devel@nongnu.org; Thu, 12 Apr 2012 21:36:51 -0400 Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id q3D1amdn023224 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Thu, 12 Apr 2012 21:36:48 -0400 Received: from amt.cnet (vpn-8-174.rdu.redhat.com [10.11.8.174]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id q3D1alXm025116; Thu, 12 Apr 2012 21:36:47 -0400 Received: from amt.cnet (amt.cnet [127.0.0.1]) by amt.cnet (Postfix) with ESMTP id F418F7A355; Thu, 12 Apr 2012 22:32:49 -0300 (BRT) Received: (from marcelo@localhost) by amt.cnet (8.14.5/8.14.5/Submit) id q3D1WljM014000; Thu, 12 Apr 2012 22:32:47 -0300 From: Marcelo Tosatti To: Anthony Liguori Date: Thu, 12 Apr 2012 22:32:36 -0300 Message-Id: In-Reply-To: References: X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 209.132.183.28 Cc: Marcelo Tosatti , qemu-devel@nongnu.org, kvm@vger.kernel.org, Stefan Hajnoczi Subject: [Qemu-devel] [PATCH 1/7] kvm: add flightrecorder script X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org From: Stefan Hajnoczi The kvm kernel module includes a number of trace events which can be useful when debugging system behavior. Even on production systems these trace events can be used to observe guest behavior and identify the source of problems. The kvm_flightrecorder script is a command-line wrapper for the /sys/kernel/debug/tracing interface. Kernel symbols do not need to be installed. This script captures a fixed-size buffer of KVM trace events. Recent events overwrite the oldest events when the buffer size is exceeded and it is possible to leave KVM tracing enabled for any period of time with just a fixed-size buffer. If the buffer is large enough this script is a useful tool for collecting detailed information after an issue occurs with a guest. Hence the name "flight recorder". The script can also be used in 'tail' mode to simply view KVM trace events as they occur. This is handy for development and to ensure that the guest is indeed running. Signed-off-by: Stefan Hajnoczi Signed-off-by: Marcelo Tosatti --- scripts/kvm/kvm_flightrecorder | 126 ++++++++++++++++++++++++++++++++++++++++ 1 files changed, 126 insertions(+), 0 deletions(-) create mode 100755 scripts/kvm/kvm_flightrecorder diff --git a/scripts/kvm/kvm_flightrecorder b/scripts/kvm/kvm_flightrecorder new file mode 100755 index 0000000..7fb1c2d --- /dev/null +++ b/scripts/kvm/kvm_flightrecorder @@ -0,0 +1,126 @@ +#!/usr/bin/env python +# +# KVM Flight Recorder - ring buffer tracing script +# +# Copyright (C) 2012 IBM Corp +# +# Author: Stefan Hajnoczi +# +# This script provides a command-line interface to kvm ftrace and is designed +# to be used as a flight recorder that is always running. To start in-memory +# recording: +# +# sudo kvm_flightrecorder start 8192 # 8 MB per-cpu ring buffers +# +# The per-cpu ring buffer size can be given in KB as an optional argument to +# the 'start' subcommand. +# +# To stop the flight recorder: +# +# sudo kvm_flightrecorder stop +# +# To dump the contents of the flight recorder (this can be done when the +# recorder is stopped or while it is running): +# +# sudo kvm_flightrecorder dump >/path/to/dump.txt +# +# To observe the trace while it is running, use the 'tail' subcommand: +# +# sudo kvm_flightrecorder tail +# +# Note that the flight recorder may impact overall system performance by +# consuming CPU cycles. No disk I/O is performed since the ring buffer holds a +# fixed-size in-memory trace. + +import sys +import os + +tracing_dir = '/sys/kernel/debug/tracing' + +def trace_path(*args): + return os.path.join(tracing_dir, *args) + +def write_file(path, data): + open(path, 'wb').write(data) + +def enable_event(subsystem, event, enable): + write_file(trace_path('events', subsystem, event, 'enable'), '1' if enable else '0') + +def enable_subsystem(subsystem, enable): + write_file(trace_path('events', subsystem, 'enable'), '1' if enable else '0') + +def start_tracing(): + enable_subsystem('kvm', True) + write_file(trace_path('tracing_on'), '1') + +def stop_tracing(): + write_file(trace_path('tracing_on'), '0') + enable_subsystem('kvm', False) + write_file(trace_path('events', 'enable'), '0') + write_file(trace_path('current_tracer'), 'nop') + +def dump_trace(): + tracefile = open(trace_path('trace'), 'r') + try: + lines = True + while lines: + lines = tracefile.readlines(64 * 1024) + sys.stdout.writelines(lines) + except KeyboardInterrupt: + pass + +def tail_trace(): + try: + for line in open(trace_path('trace_pipe'), 'r'): + sys.stdout.write(line) + except KeyboardInterrupt: + pass + +def usage(): + print 'Usage: %s start [buffer_size_kb] | stop | dump | tail' % sys.argv[0] + print 'Control the KVM flight recorder tracing.' + sys.exit(0) + +def main(): + if len(sys.argv) < 2: + usage() + + cmd = sys.argv[1] + if cmd == '--version': + print 'kvm_flightrecorder version 1.0' + sys.exit(0) + + if not os.path.isdir(tracing_dir): + print 'Unable to tracing debugfs directory, try:' + print 'mount -t debugfs none /sys/kernel/debug' + sys.exit(1) + if not os.access(tracing_dir, os.W_OK): + print 'Unable to write to tracing debugfs directory, please run as root' + sys.exit(1) + + if cmd == 'start': + stop_tracing() # clean up first + + if len(sys.argv) == 3: + try: + buffer_size_kb = int(sys.argv[2]) + except ValueError: + print 'Invalid per-cpu trace buffer size in KB' + sys.exit(1) + write_file(trace_path('buffer_size_kb'), str(buffer_size_kb)) + print 'Per-CPU ring buffer size set to %d KB' % buffer_size_kb + + start_tracing() + print 'KVM flight recorder enabled' + elif cmd == 'stop': + stop_tracing() + print 'KVM flight recorder disabled' + elif cmd == 'dump': + dump_trace() + elif cmd == 'tail': + tail_trace() + else: + usage() + +if __name__ == '__main__': + sys.exit(main())