[v2,03/11] tests/acceptance: add base class record/replay kernel tests

This patch adds a base for testing kernel boot recording and replaying.
Each test has the phase of recording and phase of replaying.
Virtual machines just boot the kernel and do not interact with
the network.
Structure and image links for the tests are borrowed from boot_linux_console.py
Testing controls the message pattern at the end of the kernel
boot for both record and replay modes. In replay mode QEMU is also
intended to finish the execution automatically.

Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>

--

v2:
 - changed default value of args to None (suggested by Willian Rampazzo)
 - inherited common functions from boot_linux_console (suggested by Willian Rampazzo)
---
 0 files changed

On 5/27/20 12:30 PM, Pavel Dovgalyuk wrote:
> This patch adds a base for testing kernel boot recording and replaying.
> Each test has the phase of recording and phase of replaying.
> Virtual machines just boot the kernel and do not interact with
> the network.
> Structure and image links for the tests are borrowed from boot_linux_console.py
> Testing controls the message pattern at the end of the kernel
> boot for both record and replay modes. In replay mode QEMU is also
> intended to finish the execution automatically.
> 
> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
> 
> --
> 
> v2:
>  - changed default value of args to None (suggested by Willian Rampazzo)
>  - inherited common functions from boot_linux_console (suggested by Willian Rampazzo)
> ---
>  0 files changed
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 47ef3139e6..e9a9ce4f66 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -2497,6 +2497,7 @@ F: net/filter-replay.c
>  F: include/sysemu/replay.h
>  F: docs/replay.txt
>  F: stubs/replay.c
> +F: tests/acceptance/replay_kernel.py
>  
>  IOVA Tree
>  M: Peter Xu <peterx@redhat.com>
> diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
> new file mode 100644
> index 0000000000..b8b277ad2f
> --- /dev/null
> +++ b/tests/acceptance/replay_kernel.py
> @@ -0,0 +1,57 @@
> +# Record/replay test that boots a Linux kernel
> +#
> +# Copyright (c) 2020 ISP RAS
> +#
> +# Author:
> +#  Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2 or
> +# later.  See the COPYING file in the top-level directory.
> +
> +import os
> +import gzip
> +
> +from avocado_qemu import wait_for_console_pattern
> +from avocado.utils import process
> +from avocado.utils import archive
> +from boot_linux_console import LinuxKernelUtils
> +
> +class ReplayKernel(LinuxKernelUtils):
> +    """
> +    Boots a Linux kernel in record mode and checks that the console
> +    is operational and the kernel command line is properly passed
> +    from QEMU to the kernel.
> +    Then replays the same scenario and verifies, that QEMU correctly
> +    terminates.
> +    """
> +
> +    timeout = 90
> +
> +    def run_vm(self, kernel_path, kernel_command_line, console_pattern,
> +               record, shift, args):
> +        vm = self.get_vm()
> +        vm.set_console()
> +        if record:
> +            mode = 'record'
> +        else:
> +            mode = 'replay'
> +        vm.add_args('-icount', 'shift=%s,rr=%s,rrfile=%s' %
> +                    (shift, mode, os.path.join(self.workdir, 'replay.bin')),

Please make replay_filename an argument, so we can add tests replaying
without having to record, and or do verification on the replay_filename
(check size, check header, run with incorrect format...).

> +                    '-kernel', kernel_path,
> +                    '-append', kernel_command_line,
> +                    '-net', 'none')
> +        if args:
> +            vm.add_args(*args)
> +        vm.launch()
> +        self.wait_for_console_pattern(console_pattern, vm)
> +        if record:
> +            vm.shutdown()
> +        else:
> +            vm.wait()
> +
> +    def run_rr(self, kernel_path, kernel_command_line, console_pattern,
> +        shift=7, args=None):
> +        self.run_vm(kernel_path, kernel_command_line, console_pattern,
> +                    True, shift, args)
> +        self.run_vm(kernel_path, kernel_command_line, console_pattern,
> +                    False, shift, args)
> 

Also please consider this snippet for logging:

-- >8 --
--- a/tests/acceptance/replay_kernel.py
+++ b/tests/acceptance/replay_kernel.py
@@ -10,6 +10,7 @@

 import os
 import gzip
+import logging

 from avocado_qemu import wait_for_console_pattern
 from avocado.utils import process
@@ -51,8 +52,11 @@ def run_vm(self, kernel_path, kernel_command_line,
console_pattern,

     def run_rr(self, kernel_path, kernel_command_line, console_pattern,
         shift=7, args=None):
+        logger = logging.getLogger('replay')
+        logger.info('recording...')
         self.run_vm(kernel_path, kernel_command_line, console_pattern,
                     True, shift, args)
+        logger.info('replaying...')
         self.run_vm(kernel_path, kernel_command_line, console_pattern,
                     False, shift, args)
---

Then you can use the logger using:

avocado --show=app,replay run -t machine:vexpress-a9 \
  tests/acceptance/replay_kernel.py

Ideally I'd like to see timestamp (Avocado surely provide the option)
and record.bin size between.

Pavel Dovgalyuk <Pavel.Dovgaluk@gmail.com> writes:

> This patch adds a base for testing kernel boot recording and replaying.
> Each test has the phase of recording and phase of replaying.
> Virtual machines just boot the kernel and do not interact with
> the network.
> Structure and image links for the tests are borrowed from boot_linux_console.py
> Testing controls the message pattern at the end of the kernel
> boot for both record and replay modes. In replay mode QEMU is also
> intended to finish the execution automatically.
>
> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>

diff --git a/MAINTAINERS b/MAINTAINERS
index 47ef3139e6..e9a9ce4f66 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2497,6 +2497,7 @@ F: net/filter-replay.c
 F: include/sysemu/replay.h
 F: docs/replay.txt
 F: stubs/replay.c
+F: tests/acceptance/replay_kernel.py
 
 IOVA Tree
 M: Peter Xu <peterx@redhat.com>
diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
new file mode 100644
index 0000000000..b8b277ad2f
--- /dev/null
+++ b/tests/acceptance/replay_kernel.py
@@ -0,0 +1,57 @@
+# Record/replay test that boots a Linux kernel
+#
+# Copyright (c) 2020 ISP RAS
+#
+# Author:
+#  Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import os
+import gzip

Do we actually use gzip in this test?

+
+from avocado_qemu import wait_for_console_pattern
+from avocado.utils import process
+from avocado.utils import archive
+from boot_linux_console import LinuxKernelUtils
+
+class ReplayKernel(LinuxKernelUtils):
+    """
+    Boots a Linux kernel in record mode and checks that the console
+    is operational and the kernel command line is properly passed
+    from QEMU to the kernel.
+    Then replays the same scenario and verifies, that QEMU correctly
+    terminates.

Shouldn't we be doing more to verify the replay behaved the same as the
recorded session? What happens if things go wrong? Does QEMU barf out or
just deviate from the previous run?

+    """
+
+    timeout = 90
+
+    def run_vm(self, kernel_path, kernel_command_line, console_pattern,
+               record, shift, args):
+        vm = self.get_vm()
+        vm.set_console()
+        if record:
+            mode = 'record'
+        else:
+            mode = 'replay'
+        vm.add_args('-icount', 'shift=%s,rr=%s,rrfile=%s' %
+                    (shift, mode, os.path.join(self.workdir, 'replay.bin')),
+                    '-kernel', kernel_path,
+                    '-append', kernel_command_line,
+                    '-net', 'none')
+        if args:
+            vm.add_args(*args)
+        vm.launch()
+        self.wait_for_console_pattern(console_pattern, vm)
+        if record:
+            vm.shutdown()
+        else:
+            vm.wait()
+
+    def run_rr(self, kernel_path, kernel_command_line, console_pattern,
+        shift=7, args=None):
+        self.run_vm(kernel_path, kernel_command_line, console_pattern,
+                    True, shift, args)
+        self.run_vm(kernel_path, kernel_command_line, console_pattern,
+                    False, shift, args)

On 27.05.2020 18:20, Alex Bennée wrote:
> Pavel Dovgalyuk <Pavel.Dovgaluk@gmail.com> writes:
>
>> This patch adds a base for testing kernel boot recording and replaying.
>> Each test has the phase of recording and phase of replaying.
>> Virtual machines just boot the kernel and do not interact with
>> the network.
>> Structure and image links for the tests are borrowed from boot_linux_console.py
>> Testing controls the message pattern at the end of the kernel
>> boot for both record and replay modes. In replay mode QEMU is also
>> intended to finish the execution automatically.
>>
>> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 47ef3139e6..e9a9ce4f66 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -2497,6 +2497,7 @@ F: net/filter-replay.c
>   F: include/sysemu/replay.h
>   F: docs/replay.txt
>   F: stubs/replay.c
> +F: tests/acceptance/replay_kernel.py
>   
>   IOVA Tree
>   M: Peter Xu <peterx@redhat.com>
> diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
> new file mode 100644
> index 0000000000..b8b277ad2f
> --- /dev/null
> +++ b/tests/acceptance/replay_kernel.py
> @@ -0,0 +1,57 @@
> +# Record/replay test that boots a Linux kernel
> +#
> +# Copyright (c) 2020 ISP RAS
> +#
> +# Author:
> +#  Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2 or
> +# later.  See the COPYING file in the top-level directory.
> +
> +import os
> +import gzip
>
> Do we actually use gzip in this test?

Removed that, thanks.

>
> +
> +from avocado_qemu import wait_for_console_pattern
> +from avocado.utils import process
> +from avocado.utils import archive
> +from boot_linux_console import LinuxKernelUtils
> +
> +class ReplayKernel(LinuxKernelUtils):
> +    """
> +    Boots a Linux kernel in record mode and checks that the console
> +    is operational and the kernel command line is properly passed
> +    from QEMU to the kernel.
> +    Then replays the same scenario and verifies, that QEMU correctly
> +    terminates.
>
> Shouldn't we be doing more to verify the replay behaved the same as the
> recorded session? What happens if things go wrong? Does QEMU barf out or
> just deviate from the previous run?

We hardly can compare vCPU states during record and replay.

But in the most cases it is not needed. When control flow goes in the 
wrong direction, it affects the interrupts and exceptions.

And interrupts and exceptions are the synchronization points in the 
replay log. Therefore when the executions differ, QEMU replay just hangs.

>
> +    """
> +
> +    timeout = 90
> +
> +    def run_vm(self, kernel_path, kernel_command_line, console_pattern,
> +               record, shift, args):
> +        vm = self.get_vm()
> +        vm.set_console()
> +        if record:
> +            mode = 'record'
> +        else:
> +            mode = 'replay'
> +        vm.add_args('-icount', 'shift=%s,rr=%s,rrfile=%s' %
> +                    (shift, mode, os.path.join(self.workdir, 'replay.bin')),
> +                    '-kernel', kernel_path,
> +                    '-append', kernel_command_line,
> +                    '-net', 'none')
> +        if args:
> +            vm.add_args(*args)
> +        vm.launch()
> +        self.wait_for_console_pattern(console_pattern, vm)
> +        if record:
> +            vm.shutdown()
> +        else:
> +            vm.wait()
> +
> +    def run_rr(self, kernel_path, kernel_command_line, console_pattern,
> +        shift=7, args=None):
> +        self.run_vm(kernel_path, kernel_command_line, console_pattern,
> +                    True, shift, args)
> +        self.run_vm(kernel_path, kernel_command_line, console_pattern,
> +                    False, shift, args)
>
>
>

Pavel Dovgalyuk <dovgaluk@ispras.ru> writes:

> On 27.05.2020 18:20, Alex Bennée wrote:
>> Pavel Dovgalyuk <Pavel.Dovgaluk@gmail.com> writes:
>>
>>> This patch adds a base for testing kernel boot recording and replaying.
>>> Each test has the phase of recording and phase of replaying.
>>> Virtual machines just boot the kernel and do not interact with
>>> the network.
>>> Structure and image links for the tests are borrowed from boot_linux_console.py
>>> Testing controls the message pattern at the end of the kernel
>>> boot for both record and replay modes. In replay mode QEMU is also
>>> intended to finish the execution automatically.
>>>
>>> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index 47ef3139e6..e9a9ce4f66 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -2497,6 +2497,7 @@ F: net/filter-replay.c
>>   F: include/sysemu/replay.h
>>   F: docs/replay.txt
>>   F: stubs/replay.c
>> +F: tests/acceptance/replay_kernel.py
>>     IOVA Tree
>>   M: Peter Xu <peterx@redhat.com>
>> diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
>> new file mode 100644
>> index 0000000000..b8b277ad2f
>> --- /dev/null
>> +++ b/tests/acceptance/replay_kernel.py
>> @@ -0,0 +1,57 @@
>> +# Record/replay test that boots a Linux kernel
>> +#
>> +# Copyright (c) 2020 ISP RAS
>> +#
>> +# Author:
>> +#  Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
>> +#
>> +# This work is licensed under the terms of the GNU GPL, version 2 or
>> +# later.  See the COPYING file in the top-level directory.
>> +
>> +import os
>> +import gzip
>>
>> Do we actually use gzip in this test?
>
> Removed that, thanks.
>
>>
>> +
>> +from avocado_qemu import wait_for_console_pattern
>> +from avocado.utils import process
>> +from avocado.utils import archive
>> +from boot_linux_console import LinuxKernelUtils
>> +
>> +class ReplayKernel(LinuxKernelUtils):
>> +    """
>> +    Boots a Linux kernel in record mode and checks that the console
>> +    is operational and the kernel command line is properly passed
>> +    from QEMU to the kernel.
>> +    Then replays the same scenario and verifies, that QEMU correctly
>> +    terminates.
>>
>> Shouldn't we be doing more to verify the replay behaved the same as the
>> recorded session? What happens if things go wrong? Does QEMU barf out or
>> just deviate from the previous run?
>
> We hardly can compare vCPU states during record and replay.
>
> But in the most cases it is not needed. When control flow goes in the
> wrong direction, it affects the interrupts and exceptions.
>
> And interrupts and exceptions are the synchronization points in the
> replay log. Therefore when the executions differ, QEMU replay just
> hangs.

Maybe we should fix that and exit with a more definitive error? Hangs
are just plain ugly to debug because your first step has to be to start
poking around with a debugger.

On 28.05.2020 11:28, Alex Bennée wrote:
> Pavel Dovgalyuk <dovgaluk@ispras.ru> writes:
>
>> On 27.05.2020 18:20, Alex Bennée wrote:
>>> Pavel Dovgalyuk <Pavel.Dovgaluk@gmail.com> writes:
>>>
>>>> This patch adds a base for testing kernel boot recording and replaying.
>>>> Each test has the phase of recording and phase of replaying.
>>>> Virtual machines just boot the kernel and do not interact with
>>>> the network.
>>>> Structure and image links for the tests are borrowed from boot_linux_console.py
>>>> Testing controls the message pattern at the end of the kernel
>>>> boot for both record and replay modes. In replay mode QEMU is also
>>>> intended to finish the execution automatically.
>>>>
>>>> Signed-off-by: Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
>>> diff --git a/MAINTAINERS b/MAINTAINERS
>>> index 47ef3139e6..e9a9ce4f66 100644
>>> --- a/MAINTAINERS
>>> +++ b/MAINTAINERS
>>> @@ -2497,6 +2497,7 @@ F: net/filter-replay.c
>>>    F: include/sysemu/replay.h
>>>    F: docs/replay.txt
>>>    F: stubs/replay.c
>>> +F: tests/acceptance/replay_kernel.py
>>>      IOVA Tree
>>>    M: Peter Xu <peterx@redhat.com>
>>> diff --git a/tests/acceptance/replay_kernel.py b/tests/acceptance/replay_kernel.py
>>> new file mode 100644
>>> index 0000000000..b8b277ad2f
>>> --- /dev/null
>>> +++ b/tests/acceptance/replay_kernel.py
>>> @@ -0,0 +1,57 @@
>>> +# Record/replay test that boots a Linux kernel
>>> +#
>>> +# Copyright (c) 2020 ISP RAS
>>> +#
>>> +# Author:
>>> +#  Pavel Dovgalyuk <Pavel.Dovgaluk@ispras.ru>
>>> +#
>>> +# This work is licensed under the terms of the GNU GPL, version 2 or
>>> +# later.  See the COPYING file in the top-level directory.
>>> +
>>> +import os
>>> +import gzip
>>>
>>> Do we actually use gzip in this test?
>> Removed that, thanks.
>>
>>> +
>>> +from avocado_qemu import wait_for_console_pattern
>>> +from avocado.utils import process
>>> +from avocado.utils import archive
>>> +from boot_linux_console import LinuxKernelUtils
>>> +
>>> +class ReplayKernel(LinuxKernelUtils):
>>> +    """
>>> +    Boots a Linux kernel in record mode and checks that the console
>>> +    is operational and the kernel command line is properly passed
>>> +    from QEMU to the kernel.
>>> +    Then replays the same scenario and verifies, that QEMU correctly
>>> +    terminates.
>>>
>>> Shouldn't we be doing more to verify the replay behaved the same as the
>>> recorded session? What happens if things go wrong? Does QEMU barf out or
>>> just deviate from the previous run?
>> We hardly can compare vCPU states during record and replay.
>>
>> But in the most cases it is not needed. When control flow goes in the
>> wrong direction, it affects the interrupts and exceptions.
>>
>> And interrupts and exceptions are the synchronization points in the
>> replay log. Therefore when the executions differ, QEMU replay just
>> hangs.
> Maybe we should fix that and exit with a more definitive error? Hangs
> are just plain ugly to debug because your first step has to be to start
> poking around with a debugger.
>
Good point, I'll thinks about it.