[4/4] test/avocado: test Linux boot up in x2APIC with userspace local APIC

Simple test to check Linux boot up in x2APIC with userspace local APIC and
TCG accelerator.

Signed-off-by: Bui Quang Minh <minhquangbui99@gmail.com>
---
 tests/avocado/tcg_x2apic.py | 91 +++++++++++++++++++++++++++++++++++++
 1 file changed, 91 insertions(+)
 create mode 100644 tests/avocado/tcg_x2apic.py

On Tue, 2023-02-21 at 23:05 +0700, Bui Quang Minh wrote:
> 
> +    def test_physical_x2apic(self):
> +        """
> +        :avocado: tags=physical_x2apic
> +        """
> +
> +        self.common_vm_setup(True)
> +
> +        self.kernel_params = (self.distro.default_kernel_params +
> +                              ' quiet intel_iommu=on x2apic_phys')
> +        self.run_and_check()
> +        self.ssh_command('dmesg | grep "Switched APIC routing to physical x2apic"')
> +
> +    def test_cluster_x2apic(self):
> +        """
> +        :avocado: tags=cluster_x2apic
> +        """
> +
> +        self.common_vm_setup(True)
> +        self.kernel_params = (self.distro.default_kernel_params +
> +                              ' quiet intel_iommu=on')
> +        self.run_and_check()
> +        self.ssh_command('dmesg | grep "Switched APIC routing to cluster x2apic"')

Shouldn't Linux also enable X2APIC in physical mode if it doesn't have
an IOMMU? It'll just refuse to online any CPUs with APIC ID > 255.

For bonus points, make the Linux guest realise that we can do those
extra 7 bits of Extended Destinatation ID, which it normally detects
from the KVM_FEATURE_MSI_EXT_DEST_ID bit under KVM.

Bui Quang Minh <minhquangbui99@gmail.com> writes:

> Simple test to check Linux boot up in x2APIC with userspace local APIC and
> TCG accelerator.

These aren't really simple tests because they are booting up a whole
distro and on my system at least they timeout:

  ➜  ./tests/venv/bin/avocado run tests/avocado/tcg_x2apic.py 
  JOB ID     : 9347a29b02111cc865ab5485ed7e06ad932420a3
  JOB LOG    : /home/alex/avocado/job-results/job-2023-03-06T22.17-9347a29/job.log
   (1/2) tests/avocado/tcg_x2apic.py:TCGx2APIC.test_physical_x2apic: INTERRUPTED: Test interrupted by SIGTERM\nRunner error occurred: Timeout reached\nOriginal status: ERROR\n
  {'name': '1-tests/avocado/tcg_x2apic.py:TCGx2APIC.test_physical_x2apic', 'logdir': '/home/alex/avocado/job-results/job-2023-03-06T22.17-9347a29/test-results/1-tests_... (120
  .67 s)
   (2/2) tests/avocado/tcg_x2apic.py:TCGx2APIC.test_cluster_x2apic: INTERRUPTED: Test interrupted by SIGTERM\nRunner error occurred: Timeout reached\nOriginal status: ERROR\n{
  'name': '2-tests/avocado/tcg_x2apic.py:TCGx2APIC.test_cluster_x2apic', 'logdir': '/home/alex/avocado/job-results/job-2023-03-06T22.17-9347a29/test-results/2-tests_a... (120.
  66 s)
  RESULTS    : PASS 0 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 2 | CANCEL 0
  JOB TIME   : 241.63 s

Most of what this exercises is already covered by the boot_linux.py
tests which I would dearly like to deprecate the TCG versions for CI
because they burn so much CI time.

Ideally what I would like are specific directed tests that exercise the
full functionality of the various emulated APICs supported by QEMU
without the weight of a full distro boot.

Do the kvm-unit-test APIC tests count? Forcing TCG seems to fail a bunch
though:

✗  env QEMU=$HOME/lsrc/qemu.git/builds/all/qemu-system-x86_64 ACCEL=tcg ./run_tests.sh -g apic
FAIL apic-split 
PASS ioapic-split (19 tests)
FAIL x2apic 
FAIL xapic

It might be the kvm-unit-tests need a slight tweak to not enable
tsc-deadline for TCG only tests?

Failing that I would accept a kernel+initrd test the focused on
exercising the various APIC paths without the weight of a full distro.

<snip>

On Tue, 21 Feb 2023 23:05:00 +0700
Bui Quang Minh <minhquangbui99@gmail.com> wrote:

> Simple test to check Linux boot up in x2APIC with userspace local APIC and
> TCG accelerator.

just an idea, while booting linux would give some coverage,
we probably would get much better coverage by using apic test case
from kvm unit test:
  https://gitlab.com/kvm-unit-tests/kvm-unit-tests/-/blob/master/x86/apic.c

> 
> Signed-off-by: Bui Quang Minh <minhquangbui99@gmail.com>
> ---
>  tests/avocado/tcg_x2apic.py | 91 +++++++++++++++++++++++++++++++++++++
>  1 file changed, 91 insertions(+)
>  create mode 100644 tests/avocado/tcg_x2apic.py
> 
> diff --git a/tests/avocado/tcg_x2apic.py b/tests/avocado/tcg_x2apic.py
> new file mode 100644
> index 0000000000..ff4f27017c
> --- /dev/null
> +++ b/tests/avocado/tcg_x2apic.py
> @@ -0,0 +1,91 @@
> +# x2APIC with TCG accelerator tests
> +# Based on intel_iommu.py, INTEL_IOMMU Functional tests
> +#
> +# Copyright (c) Bui Quang Minh <minhquangbui99@gmail.com>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2 or
> +# later. See the COPYING file in the top-level directory.
> +import os
> +
> +from avocado import skipIf
> +from avocado_qemu import LinuxTest
> +
> +@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
> +class TCGx2APIC(LinuxTest):
> +    """
> +    :avocado: tags=arch:x86_64
> +    :avocado: tags=distro:fedora
> +    :avocado: tags=distro_version:31
> +    :avocado: tags=machine:q35
> +    :avocado: tags=accel:tcg
> +    :avocado: tags=x2apic
> +    """
> +
> +    IOMMU_ADDON = ',iommu_platform=on,disable-modern=off,disable-legacy=on'
> +    kernel_path = None
> +    initrd_path = None
> +    kernel_params = None
> +
> +    def set_up_boot(self):
> +        path = self.download_boot()
> +        self.vm.add_args('-device', 'virtio-blk-pci,bus=pcie.0,scsi=off,' +
> +                         'drive=drv0,id=virtio-disk0,bootindex=1,'
> +                         'werror=stop,rerror=stop' + self.IOMMU_ADDON)
> +        self.vm.add_args('-device', 'virtio-gpu-pci' + self.IOMMU_ADDON)
> +        self.vm.add_args('-drive',
> +                         'file=%s,if=none,cache=writethrough,id=drv0' % path)
> +
> +    def setUp(self):
> +        super(TCGx2APIC, self).setUp(None, 'virtio-net-pci' + self.IOMMU_ADDON)
> +
> +    def add_common_args(self):
> +        self.vm.add_args('-device', 'virtio-rng-pci,rng=rng0')
> +        self.vm.add_args('-object',
> +                         'rng-random,id=rng0,filename=/dev/urandom')
> +
> +    def common_vm_setup(self, custom_kernel=None):
> +        self.require_accelerator('tcg')
> +        self.add_common_args()
> +        self.vm.add_args('-accel', 'tcg')
> +        self.vm.add_args('-device', 'intel-iommu,intremap=on,eim=on')
> +        self.vm.add_args('-cpu', 'qemu64,+x2apic')
> +
> +        if custom_kernel is None:
> +            return
> +
> +        kernel_url = self.distro.pxeboot_url + 'vmlinuz'
> +        initrd_url = self.distro.pxeboot_url + 'initrd.img'
> +        self.kernel_path = self.fetch_asset(kernel_url)
> +        self.initrd_path = self.fetch_asset(initrd_url)
> +
> +    def run_and_check(self):
> +        if self.kernel_path:
> +            self.vm.add_args('-kernel', self.kernel_path,
> +                             '-append', self.kernel_params,
> +                             '-initrd', self.initrd_path)
> +        self.launch_and_wait()
> +        self.ssh_command('cat /proc/cmdline')
> +        self.ssh_command('dmesg | grep "x2apic enabled"')
> +
> +    def test_physical_x2apic(self):
> +        """
> +        :avocado: tags=physical_x2apic
> +        """
> +
> +        self.common_vm_setup(True)
> +
> +        self.kernel_params = (self.distro.default_kernel_params +
> +                              ' quiet intel_iommu=on x2apic_phys')
> +        self.run_and_check()
> +        self.ssh_command('dmesg | grep "Switched APIC routing to physical x2apic"')
> +
> +    def test_cluster_x2apic(self):
> +        """
> +        :avocado: tags=cluster_x2apic
> +        """
> +
> +        self.common_vm_setup(True)
> +        self.kernel_params = (self.distro.default_kernel_params +
> +                              ' quiet intel_iommu=on')
> +        self.run_and_check()
> +        self.ssh_command('dmesg | grep "Switched APIC routing to cluster x2apic"')