Patchwork Grand Unified Tracing

login
register
mail settings
Submitter Blue Swirl
Date Aug. 23, 2011, 9:25 p.m.
Message ID <CAAu8pHt=UZNO+9pfUEqp8y2nD6xxt+x7ioaqzMjAukbhtnLNqQ@mail.gmail.com>
Download mbox | patch
Permalink /patch/111193/
State New
Headers show

Comments

Blue Swirl - Aug. 23, 2011, 9:25 p.m.
On Tue, Aug 23, 2011 at 2:54 PM, Blue Swirl <blauwirbel@gmail.com> wrote:
> On Tue, Aug 23, 2011 at 2:45 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> On Tue, Aug 23, 2011 at 3:34 PM, Blue Swirl <blauwirbel@gmail.com> wrote:
>>> On Tue, Aug 23, 2011 at 5:35 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>>>> On Mon, Aug 22, 2011 at 9:59 PM, Blue Swirl <blauwirbel@gmail.com> wrote:
>>>>> On Mon, Aug 22, 2011 at 8:23 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>>>>>> On Mon, Aug 22, 2011 at 7:14 PM, Blue Swirl <blauwirbel@gmail.com> wrote:
>>>>>>> OpenBIOS uses traditional DPRINTFs for debugging. I was considering
>>>>>>> replacing those with tracepoints, which would output to serial device
>>>>>>> or whatever DPRINTFs are using currently. This would not be extremely
>>>>>>> useful by itself, except maybe that configuration for debugging would
>>>>>>> be concentrated to single 'trace-events' file, but this would not be a
>>>>>>> major improvement over current XML configuration.
>>>>>>>
>>>>>>> But developing this further, maybe OpenBIOS could also pass the
>>>>>>> tracepoint data back to QEMU? Then all tracepoint data would be
>>>>>>> synchronized, coherent and all gathered to single place.
>>>>>>>
>>>>>>> The implementation could be that fw_cfg would be used to pass
>>>>>>> simpletrace style data. An offset should be added to event IDs and
>>>>>>> data would then be output as usual. On OpenBIOS side, the
>>>>>>> implementation would be pretty similar to current QEMU tracepoints but
>>>>>>> in place of file output there would be fw_cfg output.
>>>>>>>
>>>>>>> Syntax for trace-events file should be augmented with include
>>>>>>> directive, so that QEMU knows also OpenBIOS tracepoints. I think the
>>>>>>> only change to simpletrace.py would be to parse this directive.
>>>>>>>
>>>>>>> Controlling OpenBIOS tracepoints from QEMU monitor would be cool too.
>>>>>>>
>>>>>>> Going even further, other targets like kernels could use something
>>>>>>> similar, probably not using fw_cfg though.
>>>>>>>
>>>>>>> What do you think?
>>>>>>
>>>>>> Dhaval showed me a demo of unified host/guest Linux tracing last week.
>>>>>>  He is doing something similar except using a hypercall to pass a
>>>>>> string to the host kernel.  In his case kvm.ko handles the hypercall
>>>>>> and qemu is not involved.
>>>>>
>>>>> So the events flow directly from guest kernel to host kernel?
>>>>
>>>> Yes.
>>>
>>> How is that used then, with dmesg?
>>
>> When the host processes the trace hypercall it simply forwards the
>> trace event on to host Linux's trace.  It currently logs the string
>> from the guest and prefixes it with the guest program counter, I
>> believe.  You can dump out the trace using the usual Linux ftrace
>> debugfs files.
>>
>> But the longer term goal, I believe, is to really integrate guest and
>> host tracing so that host userspace (e.g. perf(1)) can pull out trace
>> events from the guest and just sees them tagged as belonging to a
>> particular qemu-kvm process.
>>
>>>
>>>>>> One issue with QEMU tracing is that trace call sites are static.  You
>>>>>> need to compile in a trace_*() call, which means that there are two
>>>>>> choices for how to tunnel OpenBIOS trace events:
>>>>>>
>>>>>> 1. Define a tunnel trace event:
>>>>>> openbios_event(uint64_t event_id, uint64_t arg1, uint64_t arg2, ...)
>>>>>>
>>>>>> QEMU only has one trace event to tunnel OpenBIOS trace events.  Then
>>>>>> the host is unable to pretty-print OpenBIOS traces automatically and
>>>>>> the max arguments becomes 6 - 1 (for the openbios_event tunnel event
>>>>>> id).
>>>>>>
>>>>>> 2. Generate a switch statement to demultiplex trace events:
>>>>>> void hypercall(uint64_t event_id, uint64_t arg1, ...)
>>>>>> {
>>>>>>    /* This is auto-generated by tracetool */
>>>>>>    switch (event_id) {
>>>>>>    case TRACE_EVENT_OPENBIOS_FOO:
>>>>>>        trace_openbios_foo(arg1, arg2, arg3);
>>>>>>        break;
>>>>>>    case TRACE_EVENT_OPENBIOS_BAR:
>>>>>>        trace_openbios_bar(arg1);
>>>>>>        break;
>>>>>>    ...
>>>>>>    }
>>>>>> }
>>>>>>
>>>>>> With this approach the user can toggle trace events at runtime and it
>>>>>> works out much nicer.
>>>>>
>>>>> Maybe I'm missing something, but why would we have to multiplex
>>>>> anything? Since the trace IDs are 64 bits, we can easily allocate a
>>>>> static range (starting from 0x8000000000000000) to non-native events
>>>>> so that QEMU IDs and OpenBIOS IDs do not overlap. QEMU would add this
>>>>> offset to IDs coming from fw_cfg. Then QEMU would just pass the data
>>>>> (8*64 bit words) to tracing back end. There obviously something needs
>>>>> to be changed so that OpenBIOS messages can be generated. For
>>>>> simpletrace this should be easy, for stderr and other back ends that
>>>>> may be more complicated.
>>>>
>>>> I meant that the interface generated by tracetool is a header file
>>>> with trace_*() functions.  If you want to support any tracing backend
>>>> then you need to use this interface and cannot call simpletrace.c
>>>> trace0(), trace1(), ... directly.
>>>>
>>>> Are you thinking of only supporting the simple trace backend?
>>>
>>> That would be simplest, then fw_cfg.c (or some intermediate entity)
>>> could just call trace6().
>>>
>>> But stderr would be useful too. I think that could be done by adding
>>> code to read trace-events file to grab the format string. Then a
>>> modified vfprintf would parse it but instead of va_arg(args, type) it
>>> would just grab the uint64_t arg. Your approach would be much simpler.
>>
>> I see.  I think we should avoid introducing dependencies on specific
>> trace backends, so I would go for the generated switch statement
>> above.  That way you can even use DTrace/SystemTap or other backends.
>> Also if we called traceX() directly we'd have to duplicate the logic
>> to check whether the trace event is enabled/disabled, so it becomes a
>> bit messy.
>
> I see. I'll try the switch approach then.

Here's a work in progress version for the guest tracepoints. Both QEMU
and OpenBIOS compile (stderr and simpletrace backends), but because of
the recent breakage, I couldn't test the patches. At least the event
from OpenBIOS is visible to QEMU monitor:
esp_do_command [Event ID 322] : state 1

QEMU must be configured with
'--with-guest-trace-file=/src/openbios-devel/trace-events'.

There's some danger of namespace collisions. One way to avoid those
would be that the event names could be prefixed, with prefix given
during configure time.

Patch

From 6656302a303be3360f016684db74e75f1ea3034c Mon Sep 17 00:00:00 2001
Message-Id: <6656302a303be3360f016684db74e75f1ea3034c.1314133941.git.blauwirbel@gmail.com>
From: Blue Swirl <blauwirbel@gmail.com>
Date: Tue, 23 Aug 2011 21:08:53 +0000
Subject: [PATCH] Introduce tracing system from QEMU

Instead of traditional debugging printf statements, which
have the bad habit of suffering from bit rot, introduce
tracepoints based on similar system in QEMU.

Only stderr (printk) and simpletrace (output via fw_cfg
device to QEMU) back ends are supported.

Signed-off-by: Blue Swirl <blauwirbel@gmail.com>
---
 config/examples/sparc32_config.xml |    3 +-
 config/scripts/tracetool           |  692 ++++++++++++++++++++++++++++++++++++
 drivers/esp.c                      |    3 +-
 drivers/fw_cfg.c                   |   30 ++
 include/arch/common/fw_cfg.h       |    4 +
 include/libopenbios/simpletrace.h  |   31 ++
 libopenbios/build.xml              |   33 ++
 libopenbios/simpletrace.c          |   60 +++
 trace-events                       |    1 +
 9 files changed, 855 insertions(+), 2 deletions(-)
 create mode 100644 config/scripts/tracetool
 create mode 100644 include/libopenbios/simpletrace.h
 create mode 100644 libopenbios/simpletrace.c
 create mode 100644 trace-events

diff --git a/config/examples/sparc32_config.xml b/config/examples/sparc32_config.xml
index f2d6afc..2b449cc 100644
--- a/config/examples/sparc32_config.xml
+++ b/config/examples/sparc32_config.xml
@@ -19,7 +19,8 @@ 
   <option name="CONFIG_SERIAL_PORT" type="integer" value="0"/>
   <option name="CONFIG_SERIAL_SPEED" type="integer" value="9600"/>
   <option name="CONFIG_DEBUG_OFMEM" type="boolean" value="false"/>
-
+  <option name="CONFIG_TRACE_SIMPLETRACE" type="boolean" value="true"/>
+  <option name="CONFIG_TRACE_STDERR" type="boolean" value="false"/>
 
   <!-- Module Configuration -->
   <option name="CONFIG_CMDLINE" type="boolean" value="true"/>
diff --git a/config/scripts/tracetool b/config/scripts/tracetool
new file mode 100644
index 0000000..5c0534f
--- /dev/null
+++ b/config/scripts/tracetool
@@ -0,0 +1,692 @@ 
+#!/bin/sh
+#
+# Code generator for trace events
+#
+# Copyright IBM, Corp. 2010
+#
+# This work is licensed under the terms of the GNU GPL, version 2.  See
+# the COPYING file in the top-level directory.
+
+# Disable pathname expansion, makes processing text with '*' characters simpler
+set -f
+
+usage()
+{
+    cat >&2 <<EOF
+usage: $0 [--nop | --simple | --stderr | --ust | --dtrace] [-h | -c | -d | -g]
+Generate tracing code for a file on stdin.
+
+Backends:
+  --nop     Tracing disabled
+  --simple  Simple built-in backend
+  --stderr  Stderr built-in backend
+  --ust     LTTng User Space Tracing backend
+  --dtrace  DTrace/SystemTAP backend
+
+Output formats:
+  -h     Generate .h file
+  -c     Generate .c file
+  -d     Generate .d file (DTrace only)
+  -g     Generate guest trace .c file
+  --stap Generate .stp file (DTrace with SystemTAP only)
+
+Options:
+  --binary       [path]    Full path to QEMU binary
+  --target-arch  [arch]    QEMU emulator target arch
+  --target-type  [type]    QEMU emulator target type ('system' or 'user')
+  --probe-prefix [prefix]  Prefix for dtrace probe names
+                           (default: qemu-\$targettype-\$targetarch)
+  --guest-trace            Generate guest trace function
+
+EOF
+    exit 1
+}
+
+# Get the name of a trace event
+get_name()
+{
+    echo ${1%%\(*}
+}
+
+# Get the argument list of a trace event, including types and names
+get_args()
+{
+    local args
+    args=${1#*\(}
+    args=${args%%\)*}
+    echo "$args"
+}
+
+# Get the argument name list of a trace event
+get_argnames()
+{
+    local nfields field name sep
+    nfields=0
+    sep="$2"
+    for field in $(get_args "$1"); do
+        nfields=$((nfields + 1))
+
+        # Drop pointer star
+        field=${field#\*}
+
+        # Only argument names have commas at the end
+        name=${field%,}
+        test "$field" = "$name" && continue
+
+        printf "%s%s " $name $sep
+    done
+
+    # Last argument name
+    if [ "$nfields" -gt 1 ]
+    then
+        printf "%s" "$name"
+    fi
+}
+
+# Get the number of arguments to a trace event
+get_argc()
+{
+    local name argc
+    argc=0
+    for name in $(get_argnames "$1", ","); do
+        argc=$((argc + 1))
+    done
+    echo $argc
+}
+
+# Get the format string for a trace event
+get_fmt()
+{
+    local fmt
+    fmt=${1#*\"}
+    fmt=${fmt%\"*}
+    echo "$fmt"
+}
+
+# Get the state of a trace event
+get_state()
+{
+    local str disable state
+    str=$(get_name "$1")
+    disable=${str##disable }
+    if [ "$disable" = "$str" ] ; then
+        state=1
+    else
+        state=0
+    fi
+    echo "$state"
+}
+
+linetoh_begin_nop()
+{
+    return
+}
+
+linetoh_nop()
+{
+    local name args
+    name=$(get_name "$1")
+    args=$(get_args "$1")
+
+    # Define an empty function for the trace event
+    cat <<EOF
+static inline void trace_$name($args)
+{
+}
+EOF
+}
+
+linetoh_end_nop()
+{
+    return
+}
+
+linetoc_begin_nop()
+{
+    return
+}
+
+linetoc_nop()
+{
+    # No need for function definitions in nop backend
+    return
+}
+
+linetoc_end_nop()
+{
+    return
+}
+
+linetoh_begin_simple()
+{
+    cat <<EOF
+#include "libopenbios/simpletrace.h"
+EOF
+
+    simple_event_num=0
+}
+
+cast_args_to_uint64_t()
+{
+    local arg
+    for arg in $(get_argnames "$1", ","); do
+        printf "%s" "(uint64_t)(uintptr_t)$arg"
+    done
+}
+
+linetoh_simple()
+{
+    local name args argc trace_args state
+    name=$(get_name "$1")
+    args=$(get_args "$1")
+    argc=$(get_argc "$1")
+    state=$(get_state "$1")
+    if [ "$state" = "0" ]; then
+        name=${name##disable }
+    fi
+
+    trace_args="$simple_event_num"
+    if [ "$argc" -gt 0 ]
+    then
+        trace_args="$trace_args, $(cast_args_to_uint64_t "$1")"
+    fi
+
+    cat <<EOF
+static inline void trace_$name($args)
+{
+    trace$argc($trace_args);
+}
+EOF
+
+    simple_event_num=$((simple_event_num + 1))
+}
+
+linetoh_end_simple()
+{
+    cat <<EOF
+#define NR_TRACE_EVENTS $simple_event_num
+extern TraceEvent trace_list[NR_TRACE_EVENTS];
+EOF
+}
+
+linetoc_begin_simple()
+{
+    cat <<EOF
+#include "trace.h"
+
+TraceEvent trace_list[] = {
+EOF
+    simple_event_num=0
+
+}
+
+linetoc_simple()
+{
+    local name state
+    name=$(get_name "$1")
+    state=$(get_state "$1")
+    if [ "$state" = "0" ] ; then
+        name=${name##disable }
+    fi
+    cat <<EOF
+{.tp_name = "$name", .state=$state},
+EOF
+    simple_event_num=$((simple_event_num + 1))
+}
+
+linetoc_end_simple()
+{
+    cat <<EOF
+};
+EOF
+}
+
+#STDERR
+linetoh_begin_stderr()
+{
+    cat <<EOF
+#include "libc/vsprintf.h"
+EOF
+}
+
+linetoh_stderr()
+{
+    local name args argnames argc fmt
+    name=$(get_name "$1")
+    args=$(get_args "$1")
+    argnames=$(get_argnames "$1" ",")
+    argc=$(get_argc "$1")
+    fmt=$(get_fmt "$1")
+
+    if [ "$argc" -gt 0 ]; then
+        argnames=", $argnames"
+    fi
+
+    cat <<EOF
+static inline void trace_$name($args)
+{
+    printk("$name $fmt\n" $argnames);
+}
+EOF
+}
+
+linetoh_end_stderr()
+{
+return
+}
+
+linetoc_begin_stderr()
+{
+return
+}
+
+linetoc_stderr()
+{
+return
+}
+
+linetoc_end_stderr()
+{
+return
+}
+#END OF STDERR
+
+# Clean up after UST headers which pollute the namespace
+ust_clean_namespace() {
+    cat <<EOF
+#undef mutex_lock
+#undef mutex_unlock
+#undef inline
+#undef wmb
+EOF
+}
+
+linetoh_begin_ust()
+{
+    echo "#include <ust/tracepoint.h>"
+    ust_clean_namespace
+}
+
+linetoh_ust()
+{
+    local name args argnames
+    name=$(get_name "$1")
+    args=$(get_args "$1")
+    argnames=$(get_argnames "$1", ",")
+
+    cat <<EOF
+DECLARE_TRACE(ust_$name, TP_PROTO($args), TP_ARGS($argnames));
+#define trace_$name trace_ust_$name
+EOF
+}
+
+linetoh_end_ust()
+{
+    return
+}
+
+linetoc_begin_ust()
+{
+    cat <<EOF
+#include <ust/marker.h>
+$(ust_clean_namespace)
+#include "trace.h"
+EOF
+}
+
+linetoc_ust()
+{
+    local name args argnames fmt
+    name=$(get_name "$1")
+    args=$(get_args "$1")
+    argnames=$(get_argnames "$1", ",")
+    [ -z "$argnames" ] || argnames=", $argnames"
+    fmt=$(get_fmt "$1")
+
+    cat <<EOF
+DEFINE_TRACE(ust_$name);
+
+static void ust_${name}_probe($args)
+{
+    trace_mark(ust, $name, "$fmt"$argnames);
+}
+EOF
+
+    # Collect names for later
+    names="$names $name"
+}
+
+linetoc_end_ust()
+{
+    cat <<EOF
+static void __attribute__((constructor)) trace_init(void)
+{
+EOF
+
+    for name in $names; do
+        cat <<EOF
+    register_trace_ust_$name(ust_${name}_probe);
+EOF
+    done
+
+    echo "}"
+}
+
+linetoh_begin_dtrace()
+{
+    cat <<EOF
+#include "trace-dtrace.h"
+EOF
+}
+
+linetoh_dtrace()
+{
+    local name args argnames state nameupper
+    name=$(get_name "$1")
+    args=$(get_args "$1")
+    argnames=$(get_argnames "$1", ",")
+    state=$(get_state "$1")
+    if [ "$state" = "0" ] ; then
+        name=${name##disable }
+    fi
+
+    nameupper=`echo $name | tr '[:lower:]' '[:upper:]'`
+
+    # Define an empty function for the trace event
+    cat <<EOF
+static inline void trace_$name($args) {
+    if (QEMU_${nameupper}_ENABLED()) {
+        QEMU_${nameupper}($argnames);
+    }
+}
+EOF
+}
+
+linetoh_end_dtrace()
+{
+    return
+}
+
+linetoc_begin_dtrace()
+{
+    return
+}
+
+linetoc_dtrace()
+{
+    # No need for function definitions in dtrace backend
+    return
+}
+
+linetoc_end_dtrace()
+{
+    return
+}
+
+linetod_begin_dtrace()
+{
+    cat <<EOF
+provider qemu {
+EOF
+}
+
+linetod_dtrace()
+{
+    local name args state
+    name=$(get_name "$1")
+    args=$(get_args "$1")
+    state=$(get_state "$1")
+    if [ "$state" = "0" ] ; then
+        name=${name##disable }
+    fi
+
+    # DTrace provider syntax expects foo() for empty
+    # params, not foo(void)
+    if [ "$args" = "void" ]; then
+       args=""
+    fi
+
+    # Define prototype for probe arguments
+    cat <<EOF
+        probe $name($args);
+EOF
+}
+
+linetod_end_dtrace()
+{
+    cat <<EOF
+};
+EOF
+}
+
+linetostap_begin_dtrace()
+{
+    return
+}
+
+linetostap_dtrace()
+{
+    local i arg name args arglist state
+    name=$(get_name "$1")
+    args=$(get_args "$1")
+    arglist=$(get_argnames "$1", "")
+    state=$(get_state "$1")
+    if [ "$state" = "0" ] ; then
+        name=${name##disable }
+    fi
+
+    # Define prototype for probe arguments
+    cat <<EOF
+probe $probeprefix.$name = process("$binary").mark("$name")
+{
+EOF
+
+    i=1
+    for arg in $arglist
+    do
+        # 'limit' is a reserved keyword
+        if [ "$arg" = "limit" ]; then
+          arg="_limit"
+        fi
+        cat <<EOF
+  $arg = \$arg$i;
+EOF
+        i="$((i+1))"
+    done
+
+    cat <<EOF
+}
+EOF
+}
+
+linetostap_end_dtrace()
+{
+    return
+}
+
+linetog_begin_all()
+{
+    id=0
+    cat <<EOF
+#include "trace.h"
+#include "guest-trace.h"
+
+void guest_trace(uint64_t event_id, uint64_t arg1, uint64_t arg2,
+                 uint64_t arg3, uint64_t arg4, uint64_t arg5, uint64_t arg6)
+
+{
+    switch (event_id) {
+EOF
+}
+
+linetog_all()
+{
+    local name args argc arg
+    name=$(get_name "$1")
+    argc=$(get_argc "$1")
+    fieldno=1
+    args=
+    for arg in $(get_argnames "$1", ","); do
+        args="${args}arg$fieldno"
+        [ "$fieldno" = "$argc" ] || args="$args, "
+        fieldno="$((fieldno + 1))"
+    done
+    cat <<EOF
+    case $id:
+        trace_$name($args);
+        break;
+EOF
+    id="$((id + 1))"
+}
+
+linetog_end_all()
+{
+    cat <<EOF
+    default:
+        break;
+   }
+}
+EOF
+}
+
+# Process stdin by calling begin, line, and end functions for the backend
+convert()
+{
+    local begin process_line end str disable
+    begin="lineto$1_begin_$backend"
+    process_line="lineto$1_$backend"
+    end="lineto$1_end_$backend"
+
+    "$begin"
+
+    while read -r str; do
+        # Skip comments and empty lines
+        test -z "${str%%#*}" && continue
+
+        # Process the line.  The nop backend handles disabled lines.
+        disable=${str%%disable *}
+        echo
+        if test -z "$disable"; then
+            # Pass the disabled state as an arg for the simple
+            # or DTrace backends which handle it dynamically.
+            # For all other backends, call lineto$1_nop()
+            if [ $backend = "simple" -o "$backend" = "dtrace" ]; then
+                "$process_line" "$str"
+            else
+                "lineto$1_nop" "${str##disable }"
+            fi
+        else
+            "$process_line" "$str"
+        fi
+    done
+
+    echo
+    "$end"
+}
+
+tracetoh()
+{
+    cat <<EOF
+#ifndef TRACE_H
+#define TRACE_H
+
+/* This file is autogenerated by tracetool, do not edit. */
+
+EOF
+    convert h
+    echo "#endif /* TRACE_H */"
+}
+
+tracetoc()
+{
+    echo "/* This file is autogenerated by tracetool, do not edit. */"
+    convert c
+}
+
+tracetog()
+{
+    echo "/* This file is autogenerated by tracetool, do not edit. */"
+    backend=all
+    convert g
+}
+
+tracetod()
+{
+    if [ $backend != "dtrace" ]; then
+       echo "DTrace probe generator not applicable to $backend backend"
+       exit 1
+    fi
+    echo "/* This file is autogenerated by tracetool, do not edit. */"
+    convert d
+}
+
+tracetostap()
+{
+    if [ $backend != "dtrace" ]; then
+       echo "SystemTAP tapset generator not applicable to $backend backend"
+       exit 1
+    fi
+    if [ -z "$binary" ]; then
+       echo "--binary is required for SystemTAP tapset generator"
+       exit 1
+    fi
+    if [ -z "$probeprefix" -a -z "$targettype" ]; then
+       echo "--target-type is required for SystemTAP tapset generator"
+       exit 1
+    fi
+    if [ -z "$probeprefix" -a -z "$targetarch" ]; then
+       echo "--target-arch is required for SystemTAP tapset generator"
+       exit 1
+    fi
+    if [ -z "$probeprefix" ]; then
+        probeprefix="qemu.$targettype.$targetarch";
+    fi
+    echo "/* This file is autogenerated by tracetool, do not edit. */"
+    convert stap
+}
+
+
+backend=
+output=
+binary=
+targettype=
+targetarch=
+probeprefix=
+guesttrace=
+
+
+until [ -z "$1" ]
+do
+  case "$1" in
+    "--nop" | "--simple" | "--stderr" | "--ust" | "--dtrace") backend="${1#--}" ;;
+
+    "--binary") shift ; binary="$1" ;;
+    "--target-arch") shift ; targetarch="$1" ;;
+    "--target-type") shift ; targettype="$1" ;;
+    "--probe-prefix") shift ; probeprefix="$1" ;;
+    "--guest-trace") guesttrace="yes" ;;
+
+    "-h" | "-c" | "-d" | "-g") output="${1#-}" ;;
+    "--stap") output="${1#--}" ;;
+
+    "--check-backend") exit 0 ;; # used by ./configure to test for backend
+
+    "--list-backends") # used by ./configure to list available backends
+          echo "nop simple stderr ust dtrace"
+          exit 0
+          ;;
+
+    *)
+      usage;;
+  esac
+  shift
+done
+
+if [ "$backend" = "" -o "$output" = "" ]; then
+  usage
+fi
+
+gen="traceto$output"
+"$gen"
+
+exit 0
diff --git a/drivers/esp.c b/drivers/esp.c
index 41e0394..0ab1893 100644
--- a/drivers/esp.c
+++ b/drivers/esp.c
@@ -25,6 +25,7 @@ 
 #include "asm/dma.h"
 #include "esp.h"
 #include "libopenbios/ofmem.h"
+#include "trace.h"
 
 #define BUFSIZE         4096
 
@@ -106,7 +107,7 @@  do_command(esp_private_t *esp, sd_private_t *sd, int cmdlen, int replylen)
     // Clear interrupts to avoid guests seeing spurious interrupts
     (void)esp->ll->regs[ESP_INTRPT];
 
-    DPRINTF("do_command: id %d, cmd[0] 0x%x, status 0x%x\n", sd->id, esp->buffer[0], status);
+    trace_esp_do_command(sd->id, esp->buffer[0], status);
 
     /* Target didn't want all command data? */
     if ((status & ESP_STAT_TCNT) != ESP_STAT_TCNT) {
diff --git a/drivers/fw_cfg.c b/drivers/fw_cfg.c
index 4027570..7565c96 100644
--- a/drivers/fw_cfg.c
+++ b/drivers/fw_cfg.c
@@ -18,6 +18,17 @@  fw_cfg_read(uint16_t cmd, char *buf, unsigned int nbytes)
     for (i = 0; i < nbytes; i++)
         buf[i] = *fw_cfg_data;
 }
+
+void
+fw_cfg_write(uint16_t cmd, char *buf, unsigned int nbytes)
+{
+    unsigned int i;
+
+    *fw_cfg_cmd = cmd;
+    for (i = 0; i < nbytes; i++) {
+        *fw_cfg_data = buf[i];
+    }
+}
 #else
 // XXX depends on PCI bus location, should be removed
 void
@@ -29,6 +40,17 @@  fw_cfg_read(uint16_t cmd, char *buf, unsigned int nbytes)
     for (i = 0; i < nbytes; i++)
         buf[i] = inb(CONFIG_FW_CFG_ADDR + 1);
 }
+
+void
+fw_cfg_write(uint16_t cmd, char *buf, unsigned int nbytes)
+{
+    unsigned int i;
+
+    outw(cmd, CONFIG_FW_CFG_ADDR);
+    for (i = 0; i < nbytes; i++) {
+        outb(buf[i], CONFIG_FW_CFG_ADDR + 1);
+    }
+}
 #endif
 
 uint64_t
@@ -62,6 +84,14 @@  fw_cfg_read_i16(uint16_t cmd)
 }
 
 void
+fw_cfg_write_i64(uint16_t cmd, uint64_t val)
+{
+    uint64_t buf = __cpu_to_le64(val);
+
+    fw_cfg_write(cmd, (char *)&buf, sizeof(uint64_t));
+}
+
+void
 fw_cfg_init(void)
 {
 #if defined(CONFIG_SPARC32)
diff --git a/include/arch/common/fw_cfg.h b/include/arch/common/fw_cfg.h
index b0a23cd..8f08bb7 100644
--- a/include/arch/common/fw_cfg.h
+++ b/include/arch/common/fw_cfg.h
@@ -33,6 +33,8 @@ 
 #define FW_CFG_MAX_ENTRY        (FW_CFG_FILE_FIRST+FW_CFG_FILE_SLOTS)
 
 #define FW_CFG_WRITE_CHANNEL    0x4000
+#define FW_CFG_GUEST_TRACE      (FW_CFG_WRITE_CHANNEL + 0)
+
 #define FW_CFG_ARCH_LOCAL       0x8000
 #define FW_CFG_ENTRY_MASK       ~(FW_CFG_WRITE_CHANNEL | FW_CFG_ARCH_LOCAL)
 
@@ -78,9 +80,11 @@  FWCfgState *fw_cfg_init(uint32_t ctl_port, uint32_t data_port,
 
 #ifndef NO_OPENBIOS_PROTOS
 void fw_cfg_read(uint16_t cmd, char *buf, unsigned int nbytes);
+void fw_cfg_write(uint16_t cmd, char *buf, unsigned int nbytes);
 uint64_t fw_cfg_read_i64(uint16_t cmd);
 uint32_t fw_cfg_read_i32(uint16_t cmd);
 uint16_t fw_cfg_read_i16(uint16_t cmd);
+void fw_cfg_write_i64(uint16_t cmd, uint64_t val);
 void fw_cfg_init(void);
 #endif /* NO_OPENBIOS_PROTOS */
 
diff --git a/include/libopenbios/simpletrace.h b/include/libopenbios/simpletrace.h
new file mode 100644
index 0000000..aa98ee3
--- /dev/null
+++ b/include/libopenbios/simpletrace.h
@@ -0,0 +1,31 @@ 
+/*
+ * Simple trace backend
+ *
+ * Copyright IBM, Corp. 2010
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.  See
+ * the COPYING file in the top-level directory.
+ *
+ */
+
+#ifndef SIMPLETRACE_H
+#define SIMPLETRACE_H
+
+#include "asm/types.h"
+
+typedef uint64_t TraceEventID;
+
+typedef struct {
+    const char *tp_name;
+    int state;
+} TraceEvent;
+
+void trace0(TraceEventID event);
+void trace1(TraceEventID event, uint64_t x1);
+void trace2(TraceEventID event, uint64_t x1, uint64_t x2);
+void trace3(TraceEventID event, uint64_t x1, uint64_t x2, uint64_t x3);
+void trace4(TraceEventID event, uint64_t x1, uint64_t x2, uint64_t x3, uint64_t x4);
+void trace5(TraceEventID event, uint64_t x1, uint64_t x2, uint64_t x3, uint64_t x4, uint64_t x5);
+void trace6(TraceEventID event, uint64_t x1, uint64_t x2, uint64_t x3, uint64_t x4, uint64_t x5, uint64_t x6);
+
+#endif /* SIMPLETRACE_H */
diff --git a/libopenbios/build.xml b/libopenbios/build.xml
index 04e3800..b16cd35 100644
--- a/libopenbios/build.xml
+++ b/libopenbios/build.xml
@@ -19,6 +19,39 @@ 
   <object source="linuxbios_info.c" condition="LINUXBIOS"/>
   <object source="ofmem_common.c" condition="OFMEM"/>
   <object source="xcoff_load.c" condition="LOADER_XCOFF"/>
+  <object source="simpletrace.c" condition="TRACE_SIMPLETRACE"/>
+ <executable name="target/include/trace.h" condition="TRACE_SIMPLETRACE">
+  <rule><![CDATA[ $(ODIR)/target/include/trace.h-timestamp
+
+$(ODIR)/target/include/trace.h-timestamp: $(SRCDIR)/trace-events
+	$(call quiet-command,sh $(SRCDIR)/config/scripts/tracetool --simple -h < $< > $@,"  GEN   trace.h")
+	@cmp -s $@ $(ODIR)/target/include/trace.h || cp $@ $(ODIR)/target/include/trace.h]]></rule>
+  </executable>
+ <executable name="target/libopenbios/trace.c" condition="TRACE_SIMPLETRACE">
+  <rule><![CDATA[ $(ODIR)/target/libopenbios/trace.c-timestamp
+
+$(ODIR)/target/libopenbios/trace.c-timestamp: $(SRCDIR)/trace-events
+	$(call quiet-command,sh $(SRCDIR)/config/scripts/tracetool --simple -c < $< > $@,"  GEN   trace.c")
+	@cmp -s $@ $(ODIR)/target/libopenbios/trace.c || cp $@ $(ODIR)/target/libopenbios/trace.c]]></rule>
+  </executable>
+ <executable name="target/include/trace.h" condition="TRACE_STDERR">
+  <rule><![CDATA[ $(ODIR)/target/include/trace.h-timestamp
+
+$(ODIR)/target/include/trace.h-timestamp: $(SRCDIR)/trace-events
+	$(call quiet-command,sh $(SRCDIR)/config/scripts/tracetool --stderr -h < $< > $@,"  GEN   trace.h")
+	@cmp -s $@ $(ODIR)/target/include/trace.h || cp $@ $(ODIR)/target/include/trace.h]]></rule>
+  </executable>
+ <executable name="target/libopenbios/trace.c" condition="TRACE_STDERR">
+  <rule><![CDATA[ $(ODIR)/target/libopenbios/trace.c-timestamp
+
+$(ODIR)/target/libopenbios/trace.c-timestamp: $(SRCDIR)/trace-events
+	$(call quiet-command,sh $(SRCDIR)/config/scripts/tracetool --stderr -c < $< > $@,"  GEN   trace.c")
+	@cmp -s $@ $(ODIR)/target/libopenbios/trace.c || cp $@ $(ODIR)/target/libopenbios/trace.c]]></rule>
+  </executable>
+  <executable name="target/libopenbios/trace.o">
+   <rule><![CDATA[ $(ODIR)/target/libopenbios/trace.c $(ODIR)/target/include/trace.h
+	$(call quiet-command,$(CC) $$EXTRACFLAGS $(CFLAGS) $(INCLUDES) -c -o $@ $<, "  CC    $(TARGET_DIR)$@")]]></rule>
+  </executable>
  </library>
 
  <dictionary name="openbios" target="forth">
diff --git a/libopenbios/simpletrace.c b/libopenbios/simpletrace.c
new file mode 100644
index 0000000..4fb5b2e
--- /dev/null
+++ b/libopenbios/simpletrace.c
@@ -0,0 +1,60 @@ 
+/*
+ * Simple trace backend
+ *
+ * Copyright IBM, Corp. 2010
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.  See
+ * the COPYING file in the top-level directory.
+ *
+ */
+
+#include "trace.h"
+#define NO_QEMU_PROTOS
+#include "arch/common/fw_cfg.h"
+
+static void trace(TraceEventID event, uint64_t x1, uint64_t x2, uint64_t x3,
+                  uint64_t x4, uint64_t x5, uint64_t x6)
+{
+    fw_cfg_write_i64(FW_CFG_GUEST_TRACE, event);
+    fw_cfg_write_i64(FW_CFG_GUEST_TRACE, x1);
+    fw_cfg_write_i64(FW_CFG_GUEST_TRACE, x2);
+    fw_cfg_write_i64(FW_CFG_GUEST_TRACE, x3);
+    fw_cfg_write_i64(FW_CFG_GUEST_TRACE, x4);
+    fw_cfg_write_i64(FW_CFG_GUEST_TRACE, x5);
+    fw_cfg_write_i64(FW_CFG_GUEST_TRACE, x6);
+}
+
+void trace0(TraceEventID event)
+{
+    trace(event, 0, 0, 0, 0, 0, 0);
+}
+
+void trace1(TraceEventID event, uint64_t x1)
+{
+    trace(event, x1, 0, 0, 0, 0, 0);
+}
+
+void trace2(TraceEventID event, uint64_t x1, uint64_t x2)
+{
+    trace(event, x1, x2, 0, 0, 0, 0);
+}
+
+void trace3(TraceEventID event, uint64_t x1, uint64_t x2, uint64_t x3)
+{
+    trace(event, x1, x2, x3, 0, 0, 0);
+}
+
+void trace4(TraceEventID event, uint64_t x1, uint64_t x2, uint64_t x3, uint64_t x4)
+{
+    trace(event, x1, x2, x3, x4, 0, 0);
+}
+
+void trace5(TraceEventID event, uint64_t x1, uint64_t x2, uint64_t x3, uint64_t x4, uint64_t x5)
+{
+    trace(event, x1, x2, x3, x4, x5, 0);
+}
+
+void trace6(TraceEventID event, uint64_t x1, uint64_t x2, uint64_t x3, uint64_t x4, uint64_t x5, uint64_t x6)
+{
+    trace(event, x1, x2, x3, x4, x5, x6);
+}
diff --git a/trace-events b/trace-events
new file mode 100644
index 0000000..d433b02
--- /dev/null
+++ b/trace-events
@@ -0,0 +1 @@ 
+esp_do_command(int id, int cmd, int status) "id %d, cmd[0] 0x%x, status 0x%x"
-- 
1.7.2.5