mbox series

[v3,00/19] reverse debugging

Message ID 20180523064941.26016.74274.stgit@pasha-VirtualBox
Headers show
Series reverse debugging | expand

Message

Pavel Dovgalyuk May 23, 2018, 6:49 a.m. UTC
GDB remote protocol supports reverse debugging of the targets.
It includes 'reverse step' and 'reverse continue' operations.
The first one finds the previous step of the execution,
and the second one is intended to stop at the last breakpoint that
would happen when the program is executed normally.

Reverse debugging is possible in the replay mode, when at least
one snapshot was created at the record or replay phase.
QEMU can use these snapshots for travelling back in time with GDB.

Running the execution in replay mode allows using GDB reverse debugging
commands:
 - reverse-stepi (or rsi): Steps one instruction to the past.
   QEMU loads on of the prior snapshots and proceeds to the desired
   instruction forward. When that step is reaches, execution stops.
 - reverse-continue (or rc): Runs execution "backwards".
   QEMU tries to find breakpoint or watchpoint by loaded prior snapshot
   and replaying the execution. Then QEMU loads snapshots again and
   replays to the latest breakpoint. When there are no breakpoints in
   the examined section of the execution, QEMU finds one more snapshot
   and tries again. After the first snapshot is processed, execution
   stops at this snapshot.

The set of patches include the following modifications:
 - gdbstub update for reverse debugging support
 - functions that automatically perform reverse step and reverse
   continue operations
 - hmp/qmp commands for manipulating the replay process
 - improvement of the snapshotting for saving the execution step
   in the snapshot parameters
 - other record/replay fixes

The patches are available in the repository:
https://github.com/ispras/qemu/tree/rr-180428

v3 changes:
 - Fixed PS/2 bug with save/load vm, which caused failures of the replay.
   The patch was sent separately.
 - Rebased to the new code base.
 - Minor fixes.

v2 changes:
 - documented reverse debugging
 - fixed start vmstate loading in record mode
 - documented qcow2 changes (as suggested by Eric Blake)
 - made icount SnapshotInfo field optional (as suggested by Eric Blake)
 - renamed qmp commands (as suggested by Eric Blake)
 - minor changes

---

Pavel Dovgalyuk (19):
      block: implement bdrv_snapshot_goto for blkreplay
      replay: disable default snapshot for record/replay
      replay: update docs for record/replay with block devices
      replay: don't drain/flush bdrv queue while RR is working
      replay: finish record/replay before closing the disks
      qcow2: introduce icount field for snapshots
      migration: introduce icount field for snapshots
      replay: introduce info hmp/qmp command
      replay: introduce breakpoint at the specified step
      replay: implement replay-seek command to proceed to the desired step
      replay: flush events when exiting
      timer: remove replay clock probe in deadline calculation
      replay: refine replay-time module
      translator: fix breakpoint processing
      replay: flush rr queue before loading the vmstate
      gdbstub: add reverse step support in replay mode
      gdbstub: add reverse continue support in replay mode
      replay: describe reverse debugging in docs/replay.txt
      replay: allow loading any snapshots before recording


 accel/tcg/translator.c    |    9 +
 block/blkreplay.c         |    8 +
 block/io.c                |   22 +++
 block/qapi.c              |   17 ++-
 block/qcow2-snapshot.c    |    9 +
 block/qcow2.h             |    2 
 blockdev.c                |   10 ++
 cpus.c                    |   19 ++-
 docs/interop/qcow2.txt    |    4 +
 docs/replay.txt           |   45 +++++++
 exec.c                    |    6 +
 gdbstub.c                 |   50 +++++++-
 hmp-commands-info.hx      |   14 ++
 hmp-commands.hx           |   30 +++++
 hmp.h                     |    3 
 include/block/snapshot.h  |    1 
 include/sysemu/replay.h   |   18 +++
 migration/savevm.c        |   15 +-
 qapi/block-core.json      |    5 +
 qapi/block.json           |    3 
 qapi/misc.json            |   68 +++++++++++
 replay/Makefile.objs      |    3 
 replay/replay-debugging.c |  287 +++++++++++++++++++++++++++++++++++++++++++++
 replay/replay-events.c    |   14 --
 replay/replay-internal.h  |   10 +-
 replay/replay-snapshot.c  |   17 ++-
 replay/replay-time.c      |   27 ++--
 replay/replay.c           |   22 +++
 stubs/replay.c            |   10 ++
 util/qemu-timer.c         |   11 --
 vl.c                      |   18 ++-
 31 files changed, 696 insertions(+), 81 deletions(-)
 create mode 100644 replay/replay-debugging.c

Comments

Ciro Santilli May 23, 2018, 1:23 p.m. UTC | #1
On Wed, May 23, 2018 at 7:49 AM, Pavel Dovgalyuk
<Pavel.Dovgaluk@ispras.ru> wrote:
> GDB remote protocol supports reverse debugging of the targets.
> It includes 'reverse step' and 'reverse continue' operations.
> The first one finds the previous step of the execution,
> and the second one is intended to stop at the last breakpoint that
> would happen when the program is executed normally.
>
> Reverse debugging is possible in the replay mode, when at least
> one snapshot was created at the record or replay phase.
> QEMU can use these snapshots for travelling back in time with GDB.
>
> Running the execution in replay mode allows using GDB reverse debugging
> commands:
>  - reverse-stepi (or rsi): Steps one instruction to the past.
>    QEMU loads on of the prior snapshots and proceeds to the desired
>    instruction forward. When that step is reaches, execution stops.
>  - reverse-continue (or rc): Runs execution "backwards".
>    QEMU tries to find breakpoint or watchpoint by loaded prior snapshot
>    and replaying the execution. Then QEMU loads snapshots again and
>    replays to the latest breakpoint. When there are no breakpoints in
>    the examined section of the execution, QEMU finds one more snapshot
>    and tries again. After the first snapshot is processed, execution
>    stops at this snapshot.
>
> The set of patches include the following modifications:
>  - gdbstub update for reverse debugging support
>  - functions that automatically perform reverse step and reverse
>    continue operations
>  - hmp/qmp commands for manipulating the replay process
>  - improvement of the snapshotting for saving the execution step
>    in the snapshot parameters
>  - other record/replay fixes
>
> The patches are available in the repository:
> https://github.com/ispras/qemu/tree/rr-180428
>

This branch appears to contain one month old commits, is it the correct one?

> v3 changes:
>  - Fixed PS/2 bug with save/load vm, which caused failures of the replay.
>    The patch was sent separately.
>  - Rebased to the new code base.
>  - Minor fixes.
>
> v2 changes:
>  - documented reverse debugging
>  - fixed start vmstate loading in record mode
>  - documented qcow2 changes (as suggested by Eric Blake)
>  - made icount SnapshotInfo field optional (as suggested by Eric Blake)
>  - renamed qmp commands (as suggested by Eric Blake)
>  - minor changes
>
> ---
>
> Pavel Dovgalyuk (19):
>       block: implement bdrv_snapshot_goto for blkreplay
>       replay: disable default snapshot for record/replay
>       replay: update docs for record/replay with block devices
>       replay: don't drain/flush bdrv queue while RR is working
>       replay: finish record/replay before closing the disks
>       qcow2: introduce icount field for snapshots
>       migration: introduce icount field for snapshots
>       replay: introduce info hmp/qmp command
>       replay: introduce breakpoint at the specified step
>       replay: implement replay-seek command to proceed to the desired step
>       replay: flush events when exiting
>       timer: remove replay clock probe in deadline calculation
>       replay: refine replay-time module
>       translator: fix breakpoint processing
>       replay: flush rr queue before loading the vmstate
>       gdbstub: add reverse step support in replay mode
>       gdbstub: add reverse continue support in replay mode
>       replay: describe reverse debugging in docs/replay.txt
>       replay: allow loading any snapshots before recording
>
>
>  accel/tcg/translator.c    |    9 +
>  block/blkreplay.c         |    8 +
>  block/io.c                |   22 +++
>  block/qapi.c              |   17 ++-
>  block/qcow2-snapshot.c    |    9 +
>  block/qcow2.h             |    2
>  blockdev.c                |   10 ++
>  cpus.c                    |   19 ++-
>  docs/interop/qcow2.txt    |    4 +
>  docs/replay.txt           |   45 +++++++
>  exec.c                    |    6 +
>  gdbstub.c                 |   50 +++++++-
>  hmp-commands-info.hx      |   14 ++
>  hmp-commands.hx           |   30 +++++
>  hmp.h                     |    3
>  include/block/snapshot.h  |    1
>  include/sysemu/replay.h   |   18 +++
>  migration/savevm.c        |   15 +-
>  qapi/block-core.json      |    5 +
>  qapi/block.json           |    3
>  qapi/misc.json            |   68 +++++++++++
>  replay/Makefile.objs      |    3
>  replay/replay-debugging.c |  287 +++++++++++++++++++++++++++++++++++++++++++++
>  replay/replay-events.c    |   14 --
>  replay/replay-internal.h  |   10 +-
>  replay/replay-snapshot.c  |   17 ++-
>  replay/replay-time.c      |   27 ++--
>  replay/replay.c           |   22 +++
>  stubs/replay.c            |   10 ++
>  util/qemu-timer.c         |   11 --
>  vl.c                      |   18 ++-
>  31 files changed, 696 insertions(+), 81 deletions(-)
>  create mode 100644 replay/replay-debugging.c
>
> --
> Pavel Dovgalyuk
Pavel Dovgalyuk May 23, 2018, 1:28 p.m. UTC | #2
> From: Ciro Santilli [mailto:ciro.santilli@gmail.com]
> On Wed, May 23, 2018 at 7:49 AM, Pavel Dovgalyuk
> <Pavel.Dovgaluk@ispras.ru> wrote:
> > GDB remote protocol supports reverse debugging of the targets.
> > It includes 'reverse step' and 'reverse continue' operations.
> > The first one finds the previous step of the execution,
> > and the second one is intended to stop at the last breakpoint that
> > would happen when the program is executed normally.
> >
> > Reverse debugging is possible in the replay mode, when at least
> > one snapshot was created at the record or replay phase.
> > QEMU can use these snapshots for travelling back in time with GDB.
> >
> > Running the execution in replay mode allows using GDB reverse debugging
> > commands:
> >  - reverse-stepi (or rsi): Steps one instruction to the past.
> >    QEMU loads on of the prior snapshots and proceeds to the desired
> >    instruction forward. When that step is reaches, execution stops.
> >  - reverse-continue (or rc): Runs execution "backwards".
> >    QEMU tries to find breakpoint or watchpoint by loaded prior snapshot
> >    and replaying the execution. Then QEMU loads snapshots again and
> >    replays to the latest breakpoint. When there are no breakpoints in
> >    the examined section of the execution, QEMU finds one more snapshot
> >    and tries again. After the first snapshot is processed, execution
> >    stops at this snapshot.
> >
> > The set of patches include the following modifications:
> >  - gdbstub update for reverse debugging support
> >  - functions that automatically perform reverse step and reverse
> >    continue operations
> >  - hmp/qmp commands for manipulating the replay process
> >  - improvement of the snapshotting for saving the execution step
> >    in the snapshot parameters
> >  - other record/replay fixes
> >
> > The patches are available in the repository:
> > https://github.com/ispras/qemu/tree/rr-180428
> >
> 
> This branch appears to contain one month old commits, is it the correct one?

Right.
There were no significant changes except the fix which was already queued by Paolo.
As soon as it is upstreamed, I'll update the branch.

Pavel Dovgalyuk
Ciro Santilli May 23, 2018, 3:57 p.m. UTC | #3
On Wed, May 23, 2018 at 2:28 PM, Pavel Dovgalyuk <dovgaluk@ispras.ru> wrote:
>> From: Ciro Santilli [mailto:ciro.santilli@gmail.com]
>> On Wed, May 23, 2018 at 7:49 AM, Pavel Dovgalyuk
>> <Pavel.Dovgaluk@ispras.ru> wrote:
>> > GDB remote protocol supports reverse debugging of the targets.
>> > It includes 'reverse step' and 'reverse continue' operations.
>> > The first one finds the previous step of the execution,
>> > and the second one is intended to stop at the last breakpoint that
>> > would happen when the program is executed normally.
>> >
>> > Reverse debugging is possible in the replay mode, when at least
>> > one snapshot was created at the record or replay phase.
>> > QEMU can use these snapshots for travelling back in time with GDB.
>> >
>> > Running the execution in replay mode allows using GDB reverse debugging
>> > commands:
>> >  - reverse-stepi (or rsi): Steps one instruction to the past.
>> >    QEMU loads on of the prior snapshots and proceeds to the desired
>> >    instruction forward. When that step is reaches, execution stops.
>> >  - reverse-continue (or rc): Runs execution "backwards".
>> >    QEMU tries to find breakpoint or watchpoint by loaded prior snapshot
>> >    and replaying the execution. Then QEMU loads snapshots again and
>> >    replays to the latest breakpoint. When there are no breakpoints in
>> >    the examined section of the execution, QEMU finds one more snapshot
>> >    and tries again. After the first snapshot is processed, execution
>> >    stops at this snapshot.
>> >
>> > The set of patches include the following modifications:
>> >  - gdbstub update for reverse debugging support
>> >  - functions that automatically perform reverse step and reverse
>> >    continue operations
>> >  - hmp/qmp commands for manipulating the replay process
>> >  - improvement of the snapshotting for saving the execution step
>> >    in the snapshot parameters
>> >  - other record/replay fixes
>> >
>> > The patches are available in the repository:
>> > https://github.com/ispras/qemu/tree/rr-180428
>> >
>>
>> This branch appears to contain one month old commits, is it the correct one?
>
> Right.
> There were no significant changes except the fix which was already queued by Paolo.
> As soon as it is upstreamed, I'll update the branch.
>

OK.

At the current branch 6b23df0d0ca0e5e999cd12af2b18b2a95faeb421 still
observe the same behaviour as mentioned at:
https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg04669.html

If I try to savevm during the debug replay to speedup up reverse
execution, QEMU hangs.

Have you managed to reproduce that? Or is this not an intended use
case, i.e. only savevm during record is supported?

Am I correct to understand that being able to do savevms in the middle
of a long execution is the critical feature that this adds? Otherwise
we are essentially replaying from the initial snapshot every time, so
we might as well just restart a new replay, is that true?

> Pavel Dovgalyuk
>
Pavel Dovgalyuk May 24, 2018, 6 a.m. UTC | #4
> From: Ciro Santilli [mailto:ciro.santilli@gmail.com]
> On Wed, May 23, 2018 at 2:28 PM, Pavel Dovgalyuk <dovgaluk@ispras.ru> wrote:
> >> From: Ciro Santilli [mailto:ciro.santilli@gmail.com]
> >> On Wed, May 23, 2018 at 7:49 AM, Pavel Dovgalyuk
> >> <Pavel.Dovgaluk@ispras.ru> wrote:
> >> > GDB remote protocol supports reverse debugging of the targets.
> >> > It includes 'reverse step' and 'reverse continue' operations.
> >> > The first one finds the previous step of the execution,
> >> > and the second one is intended to stop at the last breakpoint that
> >> > would happen when the program is executed normally.
> >> >
> >> > Reverse debugging is possible in the replay mode, when at least
> >> > one snapshot was created at the record or replay phase.
> >> > QEMU can use these snapshots for travelling back in time with GDB.
> >> >
> >> > Running the execution in replay mode allows using GDB reverse debugging
> >> > commands:
> >> >  - reverse-stepi (or rsi): Steps one instruction to the past.
> >> >    QEMU loads on of the prior snapshots and proceeds to the desired
> >> >    instruction forward. When that step is reaches, execution stops.
> >> >  - reverse-continue (or rc): Runs execution "backwards".
> >> >    QEMU tries to find breakpoint or watchpoint by loaded prior snapshot
> >> >    and replaying the execution. Then QEMU loads snapshots again and
> >> >    replays to the latest breakpoint. When there are no breakpoints in
> >> >    the examined section of the execution, QEMU finds one more snapshot
> >> >    and tries again. After the first snapshot is processed, execution
> >> >    stops at this snapshot.
> >> >
> >> > The set of patches include the following modifications:
> >> >  - gdbstub update for reverse debugging support
> >> >  - functions that automatically perform reverse step and reverse
> >> >    continue operations
> >> >  - hmp/qmp commands for manipulating the replay process
> >> >  - improvement of the snapshotting for saving the execution step
> >> >    in the snapshot parameters
> >> >  - other record/replay fixes
> >> >
> >> > The patches are available in the repository:
> >> > https://github.com/ispras/qemu/tree/rr-180428
> >> >
> >>
> >> This branch appears to contain one month old commits, is it the correct one?
> >
> > Right.
> > There were no significant changes except the fix which was already queued by Paolo.
> > As soon as it is upstreamed, I'll update the branch.
> >
> 
> OK.
> 
> At the current branch 6b23df0d0ca0e5e999cd12af2b18b2a95faeb421 still
> observe the same behaviour as mentioned at:
> https://lists.gnu.org/archive/html/qemu-devel/2018-04/msg04669.html
> 
> If I try to savevm during the debug replay to speedup up reverse
> execution, QEMU hangs.

This behavior was fixes by a separate patch of PS/2 controller.
Here is the branch including it: https://github.com/ispras/qemu/tree/rr-180524

> Have you managed to reproduce that? Or is this not an intended use
> case, i.e. only savevm during record is supported?

Yes, I reproduced and fixed it.
savevm should work correctly during both record and replay.

> Am I correct to understand that being able to do savevms in the middle
> of a long execution is the critical feature that this adds? Otherwise
> we are essentially replaying from the initial snapshot every time, so
> we might as well just restart a new replay, is that true?

We can start replay from any of the snapshots creating during the record and replay.

Pavel Dovgalyuk