[v3] qqq: module for synchronizing with a simulation

This patch adds an interface for pacing the execution of
QEMU to match its rtc with an external simulation clock.
Its aim is to permit QEMU to be used as a module within a
larger simulation system. This revision (v3) corrects
formatting errors detected by Patchew.

Signed-off-by: James J. Nutaro <nutarojj@ornl.gov>
---
 Makefile.target          |   3 ++
 docs/simulation-sync.txt |  60 ++++++++++++++++++++++++++
 qemu-options.hx          |  16 +++++++
 qqq.c                    | 109 +++++++++++++++++++++++++++++++++++++++++++++++
 qqq.h                    |  35 +++++++++++++++
 vl.c                     |  32 ++++++++++++++
 6 files changed, 255 insertions(+)
 create mode 100644 docs/simulation-sync.txt
 create mode 100644 qqq.c
 create mode 100644 qqq.h

On Wed,  7 Sep 2016 13:53:18 -0400
"James J. Nutaro" <nutarojj@ornl.gov> wrote:

> This patch adds an interface for pacing the execution of
> QEMU to match its rtc with an external simulation clock.
> Its aim is to permit QEMU to be used as a module within a
> larger simulation system. This revision (v3) corrects
> formatting errors detected by Patchew.
> 

The changelog should only mention what the patch actually does. If you have
to re-post newer versions before the patch is finally accepted, changes should
be mentioned...

> Signed-off-by: James J. Nutaro <nutarojj@ornl.gov>
> ---

... here, below the ---, so they don't appear in the commit changelog.

Cheers.

--
Greg

>  Makefile.target          |   3 ++
>  docs/simulation-sync.txt |  60 ++++++++++++++++++++++++++
>  qemu-options.hx          |  16 +++++++
>  qqq.c                    | 109 +++++++++++++++++++++++++++++++++++++++++++++++
>  qqq.h                    |  35 +++++++++++++++
>  vl.c                     |  32 ++++++++++++++
>  6 files changed, 255 insertions(+)
>  create mode 100644 docs/simulation-sync.txt
>  create mode 100644 qqq.c
>  create mode 100644 qqq.h
> 
> diff --git a/Makefile.target b/Makefile.target
> index 8c7a072..b7f6c4c 100644
> --- a/Makefile.target
> +++ b/Makefile.target
> @@ -103,6 +103,9 @@ obj-$(CONFIG_LIBDECNUMBER) += libdecnumber/dpd/decimal32.o
>  obj-$(CONFIG_LIBDECNUMBER) += libdecnumber/dpd/decimal64.o
>  obj-$(CONFIG_LIBDECNUMBER) += libdecnumber/dpd/decimal128.o
>  
> +# qqq
> +obj-y += qqq.o
> +
>  #########################################################
>  # Linux user emulator target
>  
> diff --git a/docs/simulation-sync.txt b/docs/simulation-sync.txt
> new file mode 100644
> index 0000000..2133093
> --- /dev/null
> +++ b/docs/simulation-sync.txt
> @@ -0,0 +1,60 @@
> += Synchronizing the virtual clock with an external source =
> +
> +QEMU has a protocol for synchronizing its virtual clock
> +with the clock of a simulator in which QEMU is embedded
> +as a component. This options is enabled with the -qqq
> +argument, and it should generally be accompanied by the
> +following additional command line arguments:
> +
> +-icount 1,sleep=off -rtc clock=vm
> +
> +The -qqq argument is used to supply file descriptors
> +for two Unix pipes. The read pipe is used by QEMU to
> +receive synchronization data from the external simulator.
> +The write pipe is used by QEMU to supply synchronization
> +data to the external emulator. The typical procedure for
> +launching QEMU in is synchronization mode has three steps:
> +
> +(1) Create two pairs of pipes with the Linux pipe function.
> +    The code segment that does this might look like
> +
> +       int pipefd1[2];
> +       int pipefd2[2];
> +       pipe(pipefd1);
> +       pipe(pipefd2);
> +
> +(2) Fork QEMU with the appropriate command line arguments.
> +    The -qqq part of the argument will look something like
> +
> +       -qqq write=pipefd1[1],read=pipefd2[0]
> +
> +(3) After forking QEMU, close pipefd1[1] and pipefd2[0].
> +    Retain the other pair of pipes for communicating with QEMU.
> +
> +The synchronization protocol is very simple. To start, the
> +external simulator writes an integer to its write pipe with
> +the amount of time in microseconds that QEMU is allowed to
> +advance. The code segment that does this might look like:
> +
> +    int ta = 1000; // Advance by 1 millisecond
> +    write(pipefd2[1],&ta,sizeof(int));
> +
> +The external simulator can then advance its clock by this
> +same amount. During this time, QEMU and the external simulator
> +will be executing in parallel. When the external simulator
> +completes its time advance, it waits for QEMU by reading from
> +its read pipe. The value read will be the actual number of
> +virtual microseconds by which QEMU has advanced its virtual clock.
> +This will be greater than or equal to the requested advance.
> +The code that does this might look like:
> +
> +   read(pipefd1[0],&ta,sizeof(int));
> +
> +These steps are repeated until either (1) the external simulator
> +closes its pipes thereby causing QEMU to terminate or (2) QEMU
> +stops executing (e.g., if the emulated computer is shutdown) and
> +causes SIGPIPE to be generated by the closing of its pipes.
> +
> +You can find an example of a simulator using this protocol in
> +the adevs simulation package at http://sourceforge.net/projects/adevs/
> +
> diff --git a/qemu-options.hx b/qemu-options.hx
> index a71aaf8..d52cc9c 100644
> --- a/qemu-options.hx
> +++ b/qemu-options.hx
> @@ -3359,6 +3359,22 @@ many timer interrupts were not processed by the Windows guest and will
>  re-inject them.
>  ETEXI
>  
> +DEF("qqq", HAS_ARG, QEMU_OPTION_qqq, \
> +    "-qqq read=fd,write=fd\n" \
> +    "                enable synchronization of the virtual clock \n" \
> +    "                with an external simulation clock\n", QEMU_ARCH_ALL)
> +STEXI
> +@item -qqq read=@var{fd0},write=@var{fd1}
> +@findex -qqq
> +Qemu will use the supplied pipes to synchronize its virtual clock with
> +an external simulation clock. Qemu will wait until a time slice size in
> +microseconds is supplied on the read pipe. Then it will execute for at
> +least that number of virtual microseconds before writing the actual
> +virtual time that has elapsed in microseconds to the write pipe. This
> +cycle will repeat until a zero is elaspsed time is requested, which
> +will cause qemu to exit.
> +ETEXI
> +
>  DEF("icount", HAS_ARG, QEMU_OPTION_icount, \
>      "-icount [shift=N|auto][,align=on|off][,sleep=on|off,rr=record|replay,rrfile=<filename>]\n" \
>      "                enable virtual instruction counter with 2^N clock ticks per\n" \
> diff --git a/qqq.c b/qqq.c
> new file mode 100644
> index 0000000..f94711f
> --- /dev/null
> +++ b/qqq.c
> @@ -0,0 +1,109 @@
> +
> +#include "qemu/osdep.h"
> +#include "qemu/timer.h"
> +#include "qemu/main-loop.h"
> +#include "sysemu/cpus.h"
> +#include "qqq.h"
> +#include <time.h>
> +
> +/* This is a Linux only feature */
> +
> +#ifndef _WIN32
> +
> +#include <unistd.h>
> +#include <assert.h>
> +
> +static int elapsed;
> +static int time_advance = -1;
> +static int read_fd = -1, write_fd = -1;
> +static int64_t t;
> +static QEMUTimer *sync_timer;
> +
> +static void cleanup_and_exit(void)
> +{
> +    close(read_fd);
> +    close(write_fd);
> +    exit(0);
> +}
> +
> +static void write_mem_value(int val)
> +{
> +    if (write(write_fd, &val, sizeof(int)) != sizeof(int)) {
> +        /* If the pipe is no good, then assume this is an
> +         * indication that we should exit.
> +         */
> +        cleanup_and_exit();
> +    }
> +}
> +
> +static int read_mem_value(void)
> +{
> +    int tmp;
> +    if (read(read_fd, &tmp, sizeof(int)) != sizeof(int)) {
> +        /* If the pipe is no good, then assume this is an
> +         * indication that we should exit.
> +         */
> +        cleanup_and_exit();
> +    }
> +    return tmp;
> +}
> +
> +static void schedule_next_event(void)
> +{
> +    /* If we got the time advance in fd_read, then don't do it
> +     * again here. */
> +    if (time_advance < 0) {
> +        /* Otherwise read the value from the pipe */
> +        time_advance = read_mem_value();
> +    }
> +    /* Schedule the next synchronization point */
> +    timer_mod(sync_timer, t + time_advance);
> +    /* Note that we need to read the time advance again on the next pass */
> +    time_advance = -1;
> +}
> +
> +static void sync_func(void *data)
> +{
> +    /* Report the actual elapsed time. */
> +    int64_t tnow = qemu_clock_get_us(QEMU_CLOCK_VIRTUAL);
> +    elapsed = tnow - t;
> +    write_mem_value(elapsed);
> +    /* Update our time of last event */
> +    t = tnow;
> +    /* Schedule the next event */
> +    schedule_next_event();
> +}
> +
> +static void fd_read(void *opaque)
> +{
> +    /* Read the time advance if its becomes available
> +     * before our timer expires */
> +    time_advance = read_mem_value();
> +}
> +
> +void setup_qqq(QemuOpts *opts)
> +{
> +    /* Initialize the simulation clock */
> +    t = 0;
> +    /* Get the communication pipes */
> +    read_fd = qemu_opt_get_number(opts, "read", 0);
> +    write_fd = qemu_opt_get_number(opts, "write", 0);
> +    /* Start the timer to ensure time warps advance the clock */
> +    sync_timer = timer_new_us(QEMU_CLOCK_VIRTUAL, sync_func, NULL);
> +    /* Get the time advance that is requested by the simulation */
> +    schedule_next_event();
> +    /* Register the file descriptor with qemu. This should ensure
> +     * the emulator doesn't pause for lack of I/O and thereby
> +     * cause the attached simulator to pause with it. */
> +    qemu_set_fd_handler(read_fd, fd_read, NULL, NULL);
> +}
> +
> +#else
> +
> +void setup_qqq(QemuOpts *opts)
> +{
> +    fprintf(stderr, "-qqq is not supported on Windows, exiting\n");
> +    exit(0);
> +}
> +
> +#endif
> diff --git a/qqq.h b/qqq.h
> new file mode 100644
> index 0000000..40a24e3
> --- /dev/null
> +++ b/qqq.h
> @@ -0,0 +1,35 @@
> +/*
> + * This work is licensed under the terms of the GNU GPL
> + * version 2. Seethe COPYING file in the top-level directory.
> + *
> + * A module for pacing the rate of advance of the computer
> + * clock in reference to an external simulation clock. The
> + * basic approach used here is adapted from QBox from Green
> + * Socs. The mode of operation is as follows:
> + *
> + * The simulator uses pipes to exchange time advance data.
> + * The external simulator starts the exchange by forking a
> + * QEMU process and passing is descriptors for a read and
> + * write pipe. Then the external simulator writes an integer
> + * (native endian) to the pipe to indicate the number of
> + * microseconds that QEMU should advance. QEMU advances its
> + * virtual clock by this amount and writes to its write pipe
> + * the actual number of microseconds that have advanced.
> + * This process continues until a pipe on either side is
> + * closed, which will either cause QEMU to exit (if the
> + * external simulator closes a pipe) or raise SIGPIPE in the
> + * external simulator (if QEMU closes a pipe).
> + *
> + * Authors:
> + *   James Nutaro <nutaro@gmail.com>
> + *
> + */
> +#ifndef QQQ_H
> +#define QQQ_H
> +
> +#include "qemu/osdep.h"
> +#include "qemu-options.h"
> +
> +void setup_qqq(QemuOpts *opts);
> +
> +#endif
> diff --git a/vl.c b/vl.c
> index ee557a1..fe72990 100644
> --- a/vl.c
> +++ b/vl.c
> @@ -122,6 +122,8 @@ int main(int argc, char **argv)
>  #include "sysemu/replay.h"
>  #include "qapi/qmp/qerror.h"
>  
> +#include "qqq.h"
> +
>  #define MAX_VIRTIO_CONSOLES 1
>  #define MAX_SCLP_CONSOLES 1
>  
> @@ -231,6 +233,23 @@ static struct {
>      { .driver = "virtio-vga",           .flag = &default_vga       },
>  };
>  
> +static QemuOptsList qemu_qqq_opts = {
> +    .name = "qqq",
> +    .implied_opt_name = "",
> +    .merge_lists = true,
> +    .head = QTAILQ_HEAD_INITIALIZER(qemu_qqq_opts.head),
> +    .desc = {
> +        {
> +            .name = "read",
> +            .type = QEMU_OPT_NUMBER,
> +        }, {
> +            .name = "write",
> +            .type = QEMU_OPT_NUMBER,
> +        },
> +        { /* end of list */ }
> +    },
> +};
> +
>  static QemuOptsList qemu_rtc_opts = {
>      .name = "rtc",
>      .head = QTAILQ_HEAD_INITIALIZER(qemu_rtc_opts.head),
> @@ -2952,6 +2971,7 @@ int main(int argc, char **argv, char **envp)
>      DisplayState *ds;
>      int cyls, heads, secs, translation;
>      QemuOpts *hda_opts = NULL, *opts, *machine_opts, *icount_opts = NULL;
> +    QemuOpts *qqq_opts = NULL;
>      QemuOptsList *olist;
>      int optind;
>      const char *optarg;
> @@ -2987,6 +3007,7 @@ int main(int argc, char **argv, char **envp)
>  
>      module_call_init(MODULE_INIT_QOM);
>  
> +    qemu_add_opts(&qemu_qqq_opts);
>      qemu_add_opts(&qemu_drive_opts);
>      qemu_add_drive_opts(&qemu_legacy_drive_opts);
>      qemu_add_drive_opts(&qemu_common_drive_opts);
> @@ -3847,6 +3868,13 @@ int main(int argc, char **argv, char **envp)
>                      exit(1);
>                  }
>                  break;
> +            case QEMU_OPTION_qqq:
> +                qqq_opts = qemu_opts_parse_noisily(qemu_find_opts("qqq"),
> +                                                      optarg, true);
> +                if (!qqq_opts) {
> +                    exit(1);
> +                }
> +                break;
>              case QEMU_OPTION_incoming:
>                  if (!incoming) {
>                      runstate_set(RUN_STATE_INMIGRATE);
> @@ -4350,6 +4378,10 @@ int main(int argc, char **argv, char **envp)
>      /* spice needs the timers to be initialized by this point */
>      qemu_spice_init();
>  
> +    if (qqq_opts) {
> +        setup_qqq(qqq_opts);
> +    }
> +
>      cpu_ticks_init();
>      if (icount_opts) {
>          if (kvm_enabled() || xen_enabled()) {

I realizes my reply wasn't sent to the forum. Thanks Greg for the feedback. I'll keep this in mind for the next revision.

Jim

Hi all,

Ping!

https://patchwork.kernel.org/patch/9319729/

Best regards,

Jim

-----Original Message-----
From: Nutaro, James J. 
Sent: Tuesday, September 20, 2016 11:23 AM
To: Greg Kurz
Cc: qemu-devel@nongnu.org
Subject: Re: [Qemu-devel] [PATCH v3] qqq: module for synchronizing with a simulation

I realizes my reply wasn't sent to the forum. Thanks Greg for the feedback. I'll keep this in mind for the next revision.

Jim