| Message ID | 20190204160325.4914-5-lersek@redhat.com |
|---|---|
| State | New |
| Headers | show |
| Series | add the BiosTablesTest UEFI app, build it with the new roms/edk2 submodule | expand |
On 2/4/19 5:03 PM, Laszlo Ersek wrote: > Introduce the following build scripts under "tests/uefi-test-tools": > > * "build.sh" builds a single module (a UEFI application) from > UefiTestToolsPkg, for a single QEMU emulation target. > > "build.sh" relies on cross-compilers when the emulation target and the > build host architecture don't match. The cross-compiler prefix is > computed according to a fixed, Linux-specific pattern. No attempt is > made to copy or reimplement the GNU Make magic from "qemu/roms/Makefile" > for cross-compiler prefix determination. The reason is that the build > host OSes that are officially supported by edk2, and those that are > supported by QEMU, intersect only in Linux. (Note that the UNIXGCC > toolchain is being removed from edk2, > <https://bugzilla.tianocore.org/show_bug.cgi?id=1377>.) > > * "Makefile" currently builds the "UefiTestToolsPkg/BiosTablesTest" > application, for arm, aarch64, i386, and x86_64, with the help of > "build.sh". > > "Makefile" turns each resultant UEFI executable into a UEFI-bootable, > qcow2-compressed ISO image. The ISO images are output as > "tests/data/uefi-boot-images/bios-tables-test.<TARGET>.iso.qcow2". > > Each ISO image should be passed to QEMU as follows: > > -drive id=boot-cd,if=none,readonly,format=qcow2,file=$ISO \ > -device virtio-scsi-pci,id=scsi0 \ > -device scsi-cd,drive=boot-cd,bus=scsi0.0,bootindex=0 \ > > "Makefile" assumes that "mkdosfs", "mtools", and "genisoimage" are > present. > > Cc: "Michael S. Tsirkin" <mst@redhat.com> > Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org> > Cc: Gerd Hoffmann <kraxel@redhat.com> > Cc: Igor Mammedov <imammedo@redhat.com> > Cc: Philippe Mathieu-Daudé <philmd@redhat.com> > Cc: Shannon Zhao <shannon.zhaosl@gmail.com> > Signed-off-by: Laszlo Ersek <lersek@redhat.com> > Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> > Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com> > --- > > Notes: > v3: > - explicitly mark the "./build.sh" recipe as recursive, with the "+" > indicator; document it in a comment [Phil] > - pick up R-b, T-b [Phil] > > v2: > - add the .NOTPARALLEL target [Phil, help-make, edk2-devel] > > tests/uefi-test-tools/Makefile | 106 ++++++++++++++ > tests/uefi-test-tools/.gitignore | 3 + > tests/uefi-test-tools/build.sh | 145 ++++++++++++++++++++ > 3 files changed, 254 insertions(+) > > diff --git a/tests/uefi-test-tools/Makefile b/tests/uefi-test-tools/Makefile > new file mode 100644 > index 000000000000..1d78bc14d51a > --- /dev/null > +++ b/tests/uefi-test-tools/Makefile > @@ -0,0 +1,106 @@ > +# Makefile for the test helper UEFI applications that run in guests. > +# > +# Copyright (C) 2019, Red Hat, Inc. > +# > +# This program and the accompanying materials are licensed and made available > +# under the terms and conditions of the BSD License that accompanies this > +# distribution. The full text of the license may be found at > +# <http://opensource.org/licenses/bsd-license.php>. > +# > +# THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS, WITHOUT > +# WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED. > + > +edk2_dir := ../../roms/edk2 > +images_dir := ../data/uefi-boot-images > +emulation_targets := arm aarch64 i386 x86_64 > +uefi_binaries := bios-tables-test > +intermediate_suffixes := .efi .fat .iso.raw > + > +images: $(foreach binary,$(uefi_binaries), \ > + $(foreach target,$(emulation_targets), \ > + $(images_dir)/$(binary).$(target).iso.qcow2)) > + > +# Preserve all intermediate targets if the build succeeds. > +# - Intermediate targets help with development & debugging. > +# - Preserving intermediate targets also keeps spurious changes out of the > +# final build products, in case the user re-runs "make" without any changes > +# to the UEFI source code. Normally, the intermediate files would have been > +# removed by the last "make" invocation, hence the re-run would rebuild them > +# from the unchanged UEFI sources. Unfortunately, the "mkdosfs" and > +# "genisoimage" utilities embed timestamp-based information in their outputs, > +# which causes git to report differences for the tracked qcow2 ISO images. > +.SECONDARY: $(foreach binary,$(uefi_binaries), \ > + $(foreach target,$(emulation_targets), \ > + $(foreach suffix,$(intermediate_suffixes), \ > + Build/$(binary).$(target)$(suffix)))) > + > +# In the pattern rules below, the stem (%, $*) stands for > +# "$(binary).$(target)". > + > +# Convert the raw ISO image to a qcow2 one, enabling compression, and using a > +# small cluster size. This allows for small binary files under git control, > +# hence for small binary patches. > +$(images_dir)/%.iso.qcow2: Build/%.iso.raw > + mkdir -p -- $(images_dir) > + $${QTEST_QEMU_IMG:-qemu-img} convert -f raw -O qcow2 -c \ > + -o cluster_size=512 -- $< $@ > + > +# Embed the "UEFI system partition" into an ISO9660 file system as an ElTorito > +# boot image. > +Build/%.iso.raw: Build/%.fat > + genisoimage -input-charset ASCII -efi-boot $(notdir $<) -no-emul-boot \ > + -quiet -o $@ -- $< > + > +# Define chained macros in order to map QEMU system emulation targets to > +# *short* UEFI architecture identifiers. Periods are allowed in, and ultimately > +# stripped from, the argument. > +map_arm_to_uefi = $(subst arm,ARM,$(1)) > +map_aarch64_to_uefi = $(subst aarch64,AA64,$(call map_arm_to_uefi,$(1))) > +map_i386_to_uefi = $(subst i386,IA32,$(call map_aarch64_to_uefi,$(1))) > +map_x86_64_to_uefi = $(subst x86_64,X64,$(call map_i386_to_uefi,$(1))) > +map_to_uefi = $(subst .,,$(call map_x86_64_to_uefi,$(1))) > + > +# Format a "UEFI system partition", using the UEFI binary as the default boot > +# loader. Add 10% size for filesystem metadata, round up to the next KB, and > +# make sure the size is large enough for a FAT filesystem. Name the filesystem > +# after the UEFI binary. (Excess characters are automatically dropped from the > +# filesystem label.) > +Build/%.fat: Build/%.efi > + rm -f -- $@ > + uefi_bin_b=$$(stat --format=%s -- $<) && \ > + uefi_fat_kb=$$(( (uefi_bin_b * 11 / 10 + 1023) / 1024 )) && \ > + uefi_fat_kb=$$(( uefi_fat_kb >= 64 ? uefi_fat_kb : 64 )) && \ > + mkdosfs -C $@ -n $(basename $(@F)) -- $$uefi_fat_kb > + MTOOLS_SKIP_CHECK=1 mmd -i $@ ::EFI > + MTOOLS_SKIP_CHECK=1 mmd -i $@ ::EFI/BOOT > + MTOOLS_SKIP_CHECK=1 mcopy -i $@ -- $< \ > + ::EFI/BOOT/BOOT$(call map_to_uefi,$(suffix $*)).EFI > + > +# In the pattern rules below, the stem (%, $*) stands for "$(target)" only. The > +# association between the UEFI binary (such as "bios-tables-test") and the > +# component name from the edk2 platform DSC file (such as "BiosTablesTest") is > +# explicit in each rule. > + > +# "build.sh" invokes the "build" utility of edk2 BaseTools. In any given edk2 > +# workspace, at most one "build" instance may be operating at a time. Therefore > +# we must serialize the rebuilding of targets in this Makefile. > +.NOTPARALLEL: > + > +# In turn, the "build" utility of edk2 BaseTools invokes another "make". > +# Although the outer "make" process advertizes its job server to all child > +# processes via MAKEFLAGS in the environment, the outer "make" closes the job > +# server file descriptors (exposed in MAKEFLAGS) before executing a recipe -- > +# unless the recipe is recognized as a recursive "make" recipe. Recipes that > +# call $(MAKE) are classified automatically as recursive; for "build.sh" below, > +# we must mark the recipe manually as recursive, by using the "+" indicator. > +# This way, when the inner "make" starts a parallel build of the target edk2 > +# module, it can communicate with the outer "make"'s job server. Clearer than the make 'POSIX jobserver interaction' man page :) > +Build/bios-tables-test.%.efi: build-edk2-tools > + +./build.sh $(edk2_dir) BiosTablesTest $* $@ > + > +build-edk2-tools: > + $(MAKE) -C $(edk2_dir)/BaseTools > + > +clean: > + rm -rf Build Conf log > + $(MAKE) -C $(edk2_dir)/BaseTools clean > diff --git a/tests/uefi-test-tools/.gitignore b/tests/uefi-test-tools/.gitignore > new file mode 100644 > index 000000000000..9f246701dea1 > --- /dev/null > +++ b/tests/uefi-test-tools/.gitignore > @@ -0,0 +1,3 @@ > +Build > +Conf > +log > diff --git a/tests/uefi-test-tools/build.sh b/tests/uefi-test-tools/build.sh > new file mode 100755 > index 000000000000..155cb75c4ddb > --- /dev/null > +++ b/tests/uefi-test-tools/build.sh > @@ -0,0 +1,145 @@ > +#!/bin/bash > + > +# Build script that determines the edk2 toolchain to use, invokes the edk2 > +# "build" utility, and copies the built UEFI binary to the requested location. > +# > +# Copyright (C) 2019, Red Hat, Inc. > +# > +# This program and the accompanying materials are licensed and made available > +# under the terms and conditions of the BSD License that accompanies this > +# distribution. The full text of the license may be found at > +# <http://opensource.org/licenses/bsd-license.php>. > +# > +# THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS, WITHOUT > +# WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED. > + > +set -e -u -C > + > +# Save the command line arguments. We need to reset $# to 0 before sourcing > +# "edksetup.sh", as it will inherit $@. > +program_name=$(basename -- "$0") > +edk2_dir=$1 > +dsc_component=$2 > +emulation_target=$3 > +uefi_binary=$4 > +shift 4 > + > +# Set up the environment for edk2 building. > +export PACKAGES_PATH=$(realpath -- "$edk2_dir") > +export WORKSPACE=$PWD > +mkdir -p Conf > + > +# Source "edksetup.sh" carefully. > +set +e +u +C > +source "$PACKAGES_PATH/edksetup.sh" > +ret=$? > +set -e -u -C > +if [ $ret -ne 0 ]; then > + exit $ret > +fi > + > +# Map the QEMU system emulation target to the following types of architecture > +# identifiers: > +# - edk2, > +# - gcc cross-compilation. > +# Cover only those targets that are supported by the UEFI spec and edk2. > +case "$emulation_target" in > + (arm) > + edk2_arch=ARM > + gcc_arch=arm > + ;; > + (aarch64) > + edk2_arch=AARCH64 > + gcc_arch=aarch64 > + ;; > + (i386) > + edk2_arch=IA32 > + gcc_arch=i686 > + ;; > + (x86_64) > + edk2_arch=X64 > + gcc_arch=x86_64 > + ;; > + (*) > + printf '%s: unknown/unsupported QEMU system emulation target "%s"\n' \ > + "$program_name" "$emulation_target" >&2 > + exit 1 > + ;; > +esac > + > +# Check if cross-compilation is needed. > +host_arch=$(uname -m) > +if [ "$gcc_arch" == "$host_arch" ] || > + ( [ "$gcc_arch" == i686 ] && [ "$host_arch" == x86_64 ] ); then > + cross_prefix= > +else > + cross_prefix=${gcc_arch}-linux-gnu- > +fi > + > +# Expose cross_prefix (which is possibly empty) to the edk2 tools. While at it, > +# determine the suitable edk2 toolchain as well. > +# - For ARM and AARCH64, edk2 only offers the GCC5 toolchain tag, which covers > +# the gcc-5+ releases. > +# - For IA32 and X64, edk2 offers the GCC44 through GCC49 toolchain tags, in > +# addition to GCC5. Unfortunately, the mapping between the toolchain tags and > +# the actual gcc releases isn't entirely trivial. Run "git-blame" on > +# "OvmfPkg/build.sh" in edk2 for more information. > +# And, because the above is too simple, we have to assign cross_prefix to an > +# edk2 build variable that is specific to both the toolchain tag and the target > +# architecture. > +case "$edk2_arch" in > + (ARM) > + edk2_toolchain=GCC5 > + export GCC5_ARM_PREFIX=$cross_prefix > + ;; > + (AARCH64) > + edk2_toolchain=GCC5 > + export GCC5_AARCH64_PREFIX=$cross_prefix > + ;; > + (IA32|X64) > + gcc_version=$("${cross_prefix}gcc" -v 2>&1 | tail -1 | awk '{print $3}') > + case "$gcc_version" in > + ([1-3].*|4.[0-3].*) > + printf '%s: unsupported gcc version "%s"\n' \ > + "$program_name" "$gcc_version" >&2 > + exit 1 > + ;; > + (4.4.*) > + edk2_toolchain=GCC44 > + ;; > + (4.5.*) > + edk2_toolchain=GCC45 > + ;; > + (4.6.*) > + edk2_toolchain=GCC46 > + ;; > + (4.7.*) > + edk2_toolchain=GCC47 > + ;; > + (4.8.*) > + edk2_toolchain=GCC48 > + ;; > + (4.9.*|6.[0-2].*) > + edk2_toolchain=GCC49 > + ;; > + (*) > + edk2_toolchain=GCC5 > + ;; > + esac > + eval "export ${edk2_toolchain}_BIN=\$cross_prefix" > + ;; > +esac > + > +# Build the UEFI binary > +mkdir -p log > +build \ > + --arch="$edk2_arch" \ > + --buildtarget=DEBUG \ > + --platform=UefiTestToolsPkg/UefiTestToolsPkg.dsc \ > + --tagname="$edk2_toolchain" \ > + --module="UefiTestToolsPkg/$dsc_component/$dsc_component.inf" \ > + --log="log/$dsc_component.$edk2_arch.log" \ > + --report-file="log/$dsc_component.$edk2_arch.report" > +cp -a -- \ > + "Build/UefiTestTools/DEBUG_${edk2_toolchain}/$edk2_arch/$dsc_component.efi" \ > + "$uefi_binary" >
On Mon, Feb 04, 2019 at 05:03:24PM +0100, Laszlo Ersek wrote: > Introduce the following build scripts under "tests/uefi-test-tools": > > * "build.sh" builds a single module (a UEFI application) from > UefiTestToolsPkg, for a single QEMU emulation target. > > "build.sh" relies on cross-compilers when the emulation target and the > build host architecture don't match. The cross-compiler prefix is > computed according to a fixed, Linux-specific pattern. No attempt is > made to copy or reimplement the GNU Make magic from "qemu/roms/Makefile" > for cross-compiler prefix determination. The reason is that the build > host OSes that are officially supported by edk2, and those that are > supported by QEMU, intersect only in Linux. (Note that the UNIXGCC > toolchain is being removed from edk2, > <https://bugzilla.tianocore.org/show_bug.cgi?id=1377>.) > > * "Makefile" currently builds the "UefiTestToolsPkg/BiosTablesTest" > application, for arm, aarch64, i386, and x86_64, with the help of > "build.sh". > > "Makefile" turns each resultant UEFI executable into a UEFI-bootable, > qcow2-compressed ISO image. The ISO images are output as > "tests/data/uefi-boot-images/bios-tables-test.<TARGET>.iso.qcow2". > > Each ISO image should be passed to QEMU as follows: > > -drive id=boot-cd,if=none,readonly,format=qcow2,file=$ISO \ > -device virtio-scsi-pci,id=scsi0 \ > -device scsi-cd,drive=boot-cd,bus=scsi0.0,bootindex=0 \ > > "Makefile" assumes that "mkdosfs", "mtools", and "genisoimage" are > present. > > Cc: "Michael S. Tsirkin" <mst@redhat.com> > Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org> > Cc: Gerd Hoffmann <kraxel@redhat.com> > Cc: Igor Mammedov <imammedo@redhat.com> > Cc: Philippe Mathieu-Daudé <philmd@redhat.com> > Cc: Shannon Zhao <shannon.zhaosl@gmail.com> > Signed-off-by: Laszlo Ersek <lersek@redhat.com> > Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> > Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com> > --- > > Notes: > v3: > - explicitly mark the "./build.sh" recipe as recursive, with the "+" > indicator; document it in a comment [Phil] > - pick up R-b, T-b [Phil] > > v2: > - add the .NOTPARALLEL target [Phil, help-make, edk2-devel] > > tests/uefi-test-tools/Makefile | 106 ++++++++++++++ > tests/uefi-test-tools/.gitignore | 3 + > tests/uefi-test-tools/build.sh | 145 ++++++++++++++++++++ > 3 files changed, 254 insertions(+) > > diff --git a/tests/uefi-test-tools/Makefile b/tests/uefi-test-tools/Makefile > new file mode 100644 > index 000000000000..1d78bc14d51a > --- /dev/null > +++ b/tests/uefi-test-tools/Makefile > @@ -0,0 +1,106 @@ > +# Makefile for the test helper UEFI applications that run in guests. > +# > +# Copyright (C) 2019, Red Hat, Inc. > +# > +# This program and the accompanying materials are licensed and made available > +# under the terms and conditions of the BSD License that accompanies this > +# distribution. The full text of the license may be found at > +# <http://opensource.org/licenses/bsd-license.php>. > +# > +# THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS, WITHOUT > +# WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED. > + > +edk2_dir := ../../roms/edk2 > +images_dir := ../data/uefi-boot-images > +emulation_targets := arm aarch64 i386 x86_64 > +uefi_binaries := bios-tables-test > +intermediate_suffixes := .efi .fat .iso.raw > + > +images: $(foreach binary,$(uefi_binaries), \ > + $(foreach target,$(emulation_targets), \ > + $(images_dir)/$(binary).$(target).iso.qcow2)) > + > +# Preserve all intermediate targets if the build succeeds. > +# - Intermediate targets help with development & debugging. > +# - Preserving intermediate targets also keeps spurious changes out of the > +# final build products, in case the user re-runs "make" without any changes > +# to the UEFI source code. Normally, the intermediate files would have been > +# removed by the last "make" invocation, hence the re-run would rebuild them > +# from the unchanged UEFI sources. Unfortunately, the "mkdosfs" and > +# "genisoimage" utilities embed timestamp-based information in their outputs, > +# which causes git to report differences for the tracked qcow2 ISO images. > +.SECONDARY: $(foreach binary,$(uefi_binaries), \ > + $(foreach target,$(emulation_targets), \ > + $(foreach suffix,$(intermediate_suffixes), \ > + Build/$(binary).$(target)$(suffix)))) > + > +# In the pattern rules below, the stem (%, $*) stands for > +# "$(binary).$(target)". > + > +# Convert the raw ISO image to a qcow2 one, enabling compression, and using a > +# small cluster size. This allows for small binary files under git control, > +# hence for small binary patches. > +$(images_dir)/%.iso.qcow2: Build/%.iso.raw > + mkdir -p -- $(images_dir) > + $${QTEST_QEMU_IMG:-qemu-img} convert -f raw -O qcow2 -c \ > + -o cluster_size=512 -- $< $@ > + > +# Embed the "UEFI system partition" into an ISO9660 file system as an ElTorito > +# boot image. > +Build/%.iso.raw: Build/%.fat > + genisoimage -input-charset ASCII -efi-boot $(notdir $<) -no-emul-boot \ > + -quiet -o $@ -- $< > + > +# Define chained macros in order to map QEMU system emulation targets to > +# *short* UEFI architecture identifiers. Periods are allowed in, and ultimately > +# stripped from, the argument. > +map_arm_to_uefi = $(subst arm,ARM,$(1)) > +map_aarch64_to_uefi = $(subst aarch64,AA64,$(call map_arm_to_uefi,$(1))) > +map_i386_to_uefi = $(subst i386,IA32,$(call map_aarch64_to_uefi,$(1))) > +map_x86_64_to_uefi = $(subst x86_64,X64,$(call map_i386_to_uefi,$(1))) > +map_to_uefi = $(subst .,,$(call map_x86_64_to_uefi,$(1))) > + > +# Format a "UEFI system partition", using the UEFI binary as the default boot > +# loader. Add 10% size for filesystem metadata, round up to the next KB, and > +# make sure the size is large enough for a FAT filesystem. Name the filesystem > +# after the UEFI binary. (Excess characters are automatically dropped from the > +# filesystem label.) > +Build/%.fat: Build/%.efi > + rm -f -- $@ > + uefi_bin_b=$$(stat --format=%s -- $<) && \ > + uefi_fat_kb=$$(( (uefi_bin_b * 11 / 10 + 1023) / 1024 )) && \ > + uefi_fat_kb=$$(( uefi_fat_kb >= 64 ? uefi_fat_kb : 64 )) && \ > + mkdosfs -C $@ -n $(basename $(@F)) -- $$uefi_fat_kb > + MTOOLS_SKIP_CHECK=1 mmd -i $@ ::EFI > + MTOOLS_SKIP_CHECK=1 mmd -i $@ ::EFI/BOOT > + MTOOLS_SKIP_CHECK=1 mcopy -i $@ -- $< \ > + ::EFI/BOOT/BOOT$(call map_to_uefi,$(suffix $*)).EFI > + > +# In the pattern rules below, the stem (%, $*) stands for "$(target)" only. The > +# association between the UEFI binary (such as "bios-tables-test") and the > +# component name from the edk2 platform DSC file (such as "BiosTablesTest") is > +# explicit in each rule. > + > +# "build.sh" invokes the "build" utility of edk2 BaseTools. In any given edk2 > +# workspace, at most one "build" instance may be operating at a time. Therefore > +# we must serialize the rebuilding of targets in this Makefile. > +.NOTPARALLEL: > + > +# In turn, the "build" utility of edk2 BaseTools invokes another "make". > +# Although the outer "make" process advertizes its job server to all child > +# processes via MAKEFLAGS in the environment, the outer "make" closes the job > +# server file descriptors (exposed in MAKEFLAGS) before executing a recipe -- > +# unless the recipe is recognized as a recursive "make" recipe. Recipes that > +# call $(MAKE) are classified automatically as recursive; for "build.sh" below, > +# we must mark the recipe manually as recursive, by using the "+" indicator. > +# This way, when the inner "make" starts a parallel build of the target edk2 > +# module, it can communicate with the outer "make"'s job server. > +Build/bios-tables-test.%.efi: build-edk2-tools > + +./build.sh $(edk2_dir) BiosTablesTest $* $@ Does this actually work with an out of tree build? Shouldn't this be SRC_PATH/tests/uefi-test-tools/ ? > + > +build-edk2-tools: > + $(MAKE) -C $(edk2_dir)/BaseTools > + > +clean: > + rm -rf Build Conf log > + $(MAKE) -C $(edk2_dir)/BaseTools clean > diff --git a/tests/uefi-test-tools/.gitignore b/tests/uefi-test-tools/.gitignore > new file mode 100644 > index 000000000000..9f246701dea1 > --- /dev/null > +++ b/tests/uefi-test-tools/.gitignore > @@ -0,0 +1,3 @@ > +Build > +Conf > +log > diff --git a/tests/uefi-test-tools/build.sh b/tests/uefi-test-tools/build.sh > new file mode 100755 > index 000000000000..155cb75c4ddb > --- /dev/null > +++ b/tests/uefi-test-tools/build.sh > @@ -0,0 +1,145 @@ > +#!/bin/bash > + > +# Build script that determines the edk2 toolchain to use, invokes the edk2 > +# "build" utility, and copies the built UEFI binary to the requested location. > +# > +# Copyright (C) 2019, Red Hat, Inc. > +# > +# This program and the accompanying materials are licensed and made available > +# under the terms and conditions of the BSD License that accompanies this > +# distribution. The full text of the license may be found at > +# <http://opensource.org/licenses/bsd-license.php>. > +# > +# THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS, WITHOUT > +# WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED. > + > +set -e -u -C > + > +# Save the command line arguments. We need to reset $# to 0 before sourcing > +# "edksetup.sh", as it will inherit $@. > +program_name=$(basename -- "$0") > +edk2_dir=$1 > +dsc_component=$2 > +emulation_target=$3 > +uefi_binary=$4 > +shift 4 > + > +# Set up the environment for edk2 building. > +export PACKAGES_PATH=$(realpath -- "$edk2_dir") > +export WORKSPACE=$PWD > +mkdir -p Conf > + > +# Source "edksetup.sh" carefully. > +set +e +u +C > +source "$PACKAGES_PATH/edksetup.sh" > +ret=$? > +set -e -u -C > +if [ $ret -ne 0 ]; then > + exit $ret > +fi > + > +# Map the QEMU system emulation target to the following types of architecture > +# identifiers: > +# - edk2, > +# - gcc cross-compilation. > +# Cover only those targets that are supported by the UEFI spec and edk2. > +case "$emulation_target" in > + (arm) > + edk2_arch=ARM > + gcc_arch=arm > + ;; > + (aarch64) > + edk2_arch=AARCH64 > + gcc_arch=aarch64 > + ;; > + (i386) > + edk2_arch=IA32 > + gcc_arch=i686 > + ;; > + (x86_64) > + edk2_arch=X64 > + gcc_arch=x86_64 > + ;; > + (*) > + printf '%s: unknown/unsupported QEMU system emulation target "%s"\n' \ > + "$program_name" "$emulation_target" >&2 > + exit 1 > + ;; > +esac > + > +# Check if cross-compilation is needed. > +host_arch=$(uname -m) > +if [ "$gcc_arch" == "$host_arch" ] || > + ( [ "$gcc_arch" == i686 ] && [ "$host_arch" == x86_64 ] ); then > + cross_prefix= > +else > + cross_prefix=${gcc_arch}-linux-gnu- > +fi > + > +# Expose cross_prefix (which is possibly empty) to the edk2 tools. While at it, > +# determine the suitable edk2 toolchain as well. > +# - For ARM and AARCH64, edk2 only offers the GCC5 toolchain tag, which covers > +# the gcc-5+ releases. > +# - For IA32 and X64, edk2 offers the GCC44 through GCC49 toolchain tags, in > +# addition to GCC5. Unfortunately, the mapping between the toolchain tags and > +# the actual gcc releases isn't entirely trivial. Run "git-blame" on > +# "OvmfPkg/build.sh" in edk2 for more information. > +# And, because the above is too simple, we have to assign cross_prefix to an > +# edk2 build variable that is specific to both the toolchain tag and the target > +# architecture. > +case "$edk2_arch" in > + (ARM) > + edk2_toolchain=GCC5 > + export GCC5_ARM_PREFIX=$cross_prefix > + ;; > + (AARCH64) > + edk2_toolchain=GCC5 > + export GCC5_AARCH64_PREFIX=$cross_prefix > + ;; > + (IA32|X64) > + gcc_version=$("${cross_prefix}gcc" -v 2>&1 | tail -1 | awk '{print $3}') > + case "$gcc_version" in > + ([1-3].*|4.[0-3].*) > + printf '%s: unsupported gcc version "%s"\n' \ > + "$program_name" "$gcc_version" >&2 > + exit 1 > + ;; > + (4.4.*) > + edk2_toolchain=GCC44 > + ;; > + (4.5.*) > + edk2_toolchain=GCC45 > + ;; > + (4.6.*) > + edk2_toolchain=GCC46 > + ;; > + (4.7.*) > + edk2_toolchain=GCC47 > + ;; > + (4.8.*) > + edk2_toolchain=GCC48 > + ;; > + (4.9.*|6.[0-2].*) > + edk2_toolchain=GCC49 > + ;; > + (*) > + edk2_toolchain=GCC5 > + ;; > + esac > + eval "export ${edk2_toolchain}_BIN=\$cross_prefix" > + ;; > +esac > + > +# Build the UEFI binary > +mkdir -p log > +build \ > + --arch="$edk2_arch" \ > + --buildtarget=DEBUG \ > + --platform=UefiTestToolsPkg/UefiTestToolsPkg.dsc \ > + --tagname="$edk2_toolchain" \ > + --module="UefiTestToolsPkg/$dsc_component/$dsc_component.inf" \ > + --log="log/$dsc_component.$edk2_arch.log" \ > + --report-file="log/$dsc_component.$edk2_arch.report" > +cp -a -- \ > + "Build/UefiTestTools/DEBUG_${edk2_toolchain}/$edk2_arch/$dsc_component.efi" \ > + "$uefi_binary" > -- > 2.19.1.3.g30247aa5d201 >
On 02/04/19 18:47, Michael S. Tsirkin wrote: > On Mon, Feb 04, 2019 at 05:03:24PM +0100, Laszlo Ersek wrote: >> Introduce the following build scripts under "tests/uefi-test-tools": >> >> * "build.sh" builds a single module (a UEFI application) from >> UefiTestToolsPkg, for a single QEMU emulation target. >> >> "build.sh" relies on cross-compilers when the emulation target and the >> build host architecture don't match. The cross-compiler prefix is >> computed according to a fixed, Linux-specific pattern. No attempt is >> made to copy or reimplement the GNU Make magic from "qemu/roms/Makefile" >> for cross-compiler prefix determination. The reason is that the build >> host OSes that are officially supported by edk2, and those that are >> supported by QEMU, intersect only in Linux. (Note that the UNIXGCC >> toolchain is being removed from edk2, >> <https://bugzilla.tianocore.org/show_bug.cgi?id=1377>.) >> >> * "Makefile" currently builds the "UefiTestToolsPkg/BiosTablesTest" >> application, for arm, aarch64, i386, and x86_64, with the help of >> "build.sh". >> >> "Makefile" turns each resultant UEFI executable into a UEFI-bootable, >> qcow2-compressed ISO image. The ISO images are output as >> "tests/data/uefi-boot-images/bios-tables-test.<TARGET>.iso.qcow2". >> >> Each ISO image should be passed to QEMU as follows: >> >> -drive id=boot-cd,if=none,readonly,format=qcow2,file=$ISO \ >> -device virtio-scsi-pci,id=scsi0 \ >> -device scsi-cd,drive=boot-cd,bus=scsi0.0,bootindex=0 \ >> >> "Makefile" assumes that "mkdosfs", "mtools", and "genisoimage" are >> present. >> >> Cc: "Michael S. Tsirkin" <mst@redhat.com> >> Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org> >> Cc: Gerd Hoffmann <kraxel@redhat.com> >> Cc: Igor Mammedov <imammedo@redhat.com> >> Cc: Philippe Mathieu-Daudé <philmd@redhat.com> >> Cc: Shannon Zhao <shannon.zhaosl@gmail.com> >> Signed-off-by: Laszlo Ersek <lersek@redhat.com> >> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> >> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com> >> --- >> >> Notes: >> v3: >> - explicitly mark the "./build.sh" recipe as recursive, with the "+" >> indicator; document it in a comment [Phil] >> - pick up R-b, T-b [Phil] >> >> v2: >> - add the .NOTPARALLEL target [Phil, help-make, edk2-devel] >> >> tests/uefi-test-tools/Makefile | 106 ++++++++++++++ >> tests/uefi-test-tools/.gitignore | 3 + >> tests/uefi-test-tools/build.sh | 145 ++++++++++++++++++++ >> 3 files changed, 254 insertions(+) >> >> diff --git a/tests/uefi-test-tools/Makefile b/tests/uefi-test-tools/Makefile >> new file mode 100644 >> index 000000000000..1d78bc14d51a >> --- /dev/null >> +++ b/tests/uefi-test-tools/Makefile >> @@ -0,0 +1,106 @@ >> +# Makefile for the test helper UEFI applications that run in guests. >> +# >> +# Copyright (C) 2019, Red Hat, Inc. >> +# >> +# This program and the accompanying materials are licensed and made available >> +# under the terms and conditions of the BSD License that accompanies this >> +# distribution. The full text of the license may be found at >> +# <http://opensource.org/licenses/bsd-license.php>. >> +# >> +# THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS, WITHOUT >> +# WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED. >> + >> +edk2_dir := ../../roms/edk2 >> +images_dir := ../data/uefi-boot-images >> +emulation_targets := arm aarch64 i386 x86_64 >> +uefi_binaries := bios-tables-test >> +intermediate_suffixes := .efi .fat .iso.raw >> + >> +images: $(foreach binary,$(uefi_binaries), \ >> + $(foreach target,$(emulation_targets), \ >> + $(images_dir)/$(binary).$(target).iso.qcow2)) >> + >> +# Preserve all intermediate targets if the build succeeds. >> +# - Intermediate targets help with development & debugging. >> +# - Preserving intermediate targets also keeps spurious changes out of the >> +# final build products, in case the user re-runs "make" without any changes >> +# to the UEFI source code. Normally, the intermediate files would have been >> +# removed by the last "make" invocation, hence the re-run would rebuild them >> +# from the unchanged UEFI sources. Unfortunately, the "mkdosfs" and >> +# "genisoimage" utilities embed timestamp-based information in their outputs, >> +# which causes git to report differences for the tracked qcow2 ISO images. >> +.SECONDARY: $(foreach binary,$(uefi_binaries), \ >> + $(foreach target,$(emulation_targets), \ >> + $(foreach suffix,$(intermediate_suffixes), \ >> + Build/$(binary).$(target)$(suffix)))) >> + >> +# In the pattern rules below, the stem (%, $*) stands for >> +# "$(binary).$(target)". >> + >> +# Convert the raw ISO image to a qcow2 one, enabling compression, and using a >> +# small cluster size. This allows for small binary files under git control, >> +# hence for small binary patches. >> +$(images_dir)/%.iso.qcow2: Build/%.iso.raw >> + mkdir -p -- $(images_dir) >> + $${QTEST_QEMU_IMG:-qemu-img} convert -f raw -O qcow2 -c \ >> + -o cluster_size=512 -- $< $@ >> + >> +# Embed the "UEFI system partition" into an ISO9660 file system as an ElTorito >> +# boot image. >> +Build/%.iso.raw: Build/%.fat >> + genisoimage -input-charset ASCII -efi-boot $(notdir $<) -no-emul-boot \ >> + -quiet -o $@ -- $< >> + >> +# Define chained macros in order to map QEMU system emulation targets to >> +# *short* UEFI architecture identifiers. Periods are allowed in, and ultimately >> +# stripped from, the argument. >> +map_arm_to_uefi = $(subst arm,ARM,$(1)) >> +map_aarch64_to_uefi = $(subst aarch64,AA64,$(call map_arm_to_uefi,$(1))) >> +map_i386_to_uefi = $(subst i386,IA32,$(call map_aarch64_to_uefi,$(1))) >> +map_x86_64_to_uefi = $(subst x86_64,X64,$(call map_i386_to_uefi,$(1))) >> +map_to_uefi = $(subst .,,$(call map_x86_64_to_uefi,$(1))) >> + >> +# Format a "UEFI system partition", using the UEFI binary as the default boot >> +# loader. Add 10% size for filesystem metadata, round up to the next KB, and >> +# make sure the size is large enough for a FAT filesystem. Name the filesystem >> +# after the UEFI binary. (Excess characters are automatically dropped from the >> +# filesystem label.) >> +Build/%.fat: Build/%.efi >> + rm -f -- $@ >> + uefi_bin_b=$$(stat --format=%s -- $<) && \ >> + uefi_fat_kb=$$(( (uefi_bin_b * 11 / 10 + 1023) / 1024 )) && \ >> + uefi_fat_kb=$$(( uefi_fat_kb >= 64 ? uefi_fat_kb : 64 )) && \ >> + mkdosfs -C $@ -n $(basename $(@F)) -- $$uefi_fat_kb >> + MTOOLS_SKIP_CHECK=1 mmd -i $@ ::EFI >> + MTOOLS_SKIP_CHECK=1 mmd -i $@ ::EFI/BOOT >> + MTOOLS_SKIP_CHECK=1 mcopy -i $@ -- $< \ >> + ::EFI/BOOT/BOOT$(call map_to_uefi,$(suffix $*)).EFI >> + >> +# In the pattern rules below, the stem (%, $*) stands for "$(target)" only. The >> +# association between the UEFI binary (such as "bios-tables-test") and the >> +# component name from the edk2 platform DSC file (such as "BiosTablesTest") is >> +# explicit in each rule. >> + >> +# "build.sh" invokes the "build" utility of edk2 BaseTools. In any given edk2 >> +# workspace, at most one "build" instance may be operating at a time. Therefore >> +# we must serialize the rebuilding of targets in this Makefile. >> +.NOTPARALLEL: >> + >> +# In turn, the "build" utility of edk2 BaseTools invokes another "make". >> +# Although the outer "make" process advertizes its job server to all child >> +# processes via MAKEFLAGS in the environment, the outer "make" closes the job >> +# server file descriptors (exposed in MAKEFLAGS) before executing a recipe -- >> +# unless the recipe is recognized as a recursive "make" recipe. Recipes that >> +# call $(MAKE) are classified automatically as recursive; for "build.sh" below, >> +# we must mark the recipe manually as recursive, by using the "+" indicator. >> +# This way, when the inner "make" starts a parallel build of the target edk2 >> +# module, it can communicate with the outer "make"'s job server. >> +Build/bios-tables-test.%.efi: build-edk2-tools >> + +./build.sh $(edk2_dir) BiosTablesTest $* $@ > > Does this actually work with an out of tree build? It's not supposed to. Again, it's not something that a normal QEMU build includes. It is only for maintainers to rebuild when there is a reason to do so. The output binaries are tracked by git, and will be used as-is (in binary form) by the ACPI test suite. If there are updates to the UEFI source code, the binaries will have to be rebuilt by a maintainer (or by me, if I submit the UEFI code changes), and the refreshed blobs are to be checked into git. Think iPXE oproms for an analogy. > Shouldn't this be SRC_PATH/tests/uefi-test-tools/ ? No; nothing under roms/ is built like that, and the same applies to this patch as well. *Conceptually*, this patch is for roms/. However, in earlier discussion, it was suggested that roms/ be kept dedicated to external git submodules only, and that we not add such ROM source to roms/ whose master repo is genuinely the QEMU repo. Please see the sub-thread at: Re: [PATCH 10/14] tests: acpi: ignore SMBIOS tests when UEFI firmware is used http://mid.mail-archive.com/20190116115217.jduhqrwbjhuibmoq@sirius.home.kraxel.org The last idea was that the UEFI source code should be kept in a direct subdirectory of tests/ (rather than in roms/). And the binaries should go under tests/data/uefi-boot-images/ (rather than pc-bios/). Thanks Laszlo
On Mon, Feb 04, 2019 at 07:41:38PM +0100, Laszlo Ersek wrote: > On 02/04/19 18:47, Michael S. Tsirkin wrote: > > On Mon, Feb 04, 2019 at 05:03:24PM +0100, Laszlo Ersek wrote: > >> Introduce the following build scripts under "tests/uefi-test-tools": > >> > >> * "build.sh" builds a single module (a UEFI application) from > >> UefiTestToolsPkg, for a single QEMU emulation target. > >> > >> "build.sh" relies on cross-compilers when the emulation target and the > >> build host architecture don't match. The cross-compiler prefix is > >> computed according to a fixed, Linux-specific pattern. No attempt is > >> made to copy or reimplement the GNU Make magic from "qemu/roms/Makefile" > >> for cross-compiler prefix determination. The reason is that the build > >> host OSes that are officially supported by edk2, and those that are > >> supported by QEMU, intersect only in Linux. (Note that the UNIXGCC > >> toolchain is being removed from edk2, > >> <https://bugzilla.tianocore.org/show_bug.cgi?id=1377>.) > >> > >> * "Makefile" currently builds the "UefiTestToolsPkg/BiosTablesTest" > >> application, for arm, aarch64, i386, and x86_64, with the help of > >> "build.sh". > >> > >> "Makefile" turns each resultant UEFI executable into a UEFI-bootable, > >> qcow2-compressed ISO image. The ISO images are output as > >> "tests/data/uefi-boot-images/bios-tables-test.<TARGET>.iso.qcow2". > >> > >> Each ISO image should be passed to QEMU as follows: > >> > >> -drive id=boot-cd,if=none,readonly,format=qcow2,file=$ISO \ > >> -device virtio-scsi-pci,id=scsi0 \ > >> -device scsi-cd,drive=boot-cd,bus=scsi0.0,bootindex=0 \ > >> > >> "Makefile" assumes that "mkdosfs", "mtools", and "genisoimage" are > >> present. > >> > >> Cc: "Michael S. Tsirkin" <mst@redhat.com> > >> Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org> > >> Cc: Gerd Hoffmann <kraxel@redhat.com> > >> Cc: Igor Mammedov <imammedo@redhat.com> > >> Cc: Philippe Mathieu-Daudé <philmd@redhat.com> > >> Cc: Shannon Zhao <shannon.zhaosl@gmail.com> > >> Signed-off-by: Laszlo Ersek <lersek@redhat.com> > >> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> > >> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com> > >> --- > >> > >> Notes: > >> v3: > >> - explicitly mark the "./build.sh" recipe as recursive, with the "+" > >> indicator; document it in a comment [Phil] > >> - pick up R-b, T-b [Phil] > >> > >> v2: > >> - add the .NOTPARALLEL target [Phil, help-make, edk2-devel] > >> > >> tests/uefi-test-tools/Makefile | 106 ++++++++++++++ > >> tests/uefi-test-tools/.gitignore | 3 + > >> tests/uefi-test-tools/build.sh | 145 ++++++++++++++++++++ > >> 3 files changed, 254 insertions(+) > >> > >> diff --git a/tests/uefi-test-tools/Makefile b/tests/uefi-test-tools/Makefile > >> new file mode 100644 > >> index 000000000000..1d78bc14d51a > >> --- /dev/null > >> +++ b/tests/uefi-test-tools/Makefile > >> @@ -0,0 +1,106 @@ > >> +# Makefile for the test helper UEFI applications that run in guests. > >> +# > >> +# Copyright (C) 2019, Red Hat, Inc. > >> +# > >> +# This program and the accompanying materials are licensed and made available > >> +# under the terms and conditions of the BSD License that accompanies this > >> +# distribution. The full text of the license may be found at > >> +# <http://opensource.org/licenses/bsd-license.php>. > >> +# > >> +# THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS, WITHOUT > >> +# WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED. > >> + > >> +edk2_dir := ../../roms/edk2 > >> +images_dir := ../data/uefi-boot-images > >> +emulation_targets := arm aarch64 i386 x86_64 > >> +uefi_binaries := bios-tables-test > >> +intermediate_suffixes := .efi .fat .iso.raw > >> + > >> +images: $(foreach binary,$(uefi_binaries), \ > >> + $(foreach target,$(emulation_targets), \ > >> + $(images_dir)/$(binary).$(target).iso.qcow2)) > >> + > >> +# Preserve all intermediate targets if the build succeeds. > >> +# - Intermediate targets help with development & debugging. > >> +# - Preserving intermediate targets also keeps spurious changes out of the > >> +# final build products, in case the user re-runs "make" without any changes > >> +# to the UEFI source code. Normally, the intermediate files would have been > >> +# removed by the last "make" invocation, hence the re-run would rebuild them > >> +# from the unchanged UEFI sources. Unfortunately, the "mkdosfs" and > >> +# "genisoimage" utilities embed timestamp-based information in their outputs, > >> +# which causes git to report differences for the tracked qcow2 ISO images. > >> +.SECONDARY: $(foreach binary,$(uefi_binaries), \ > >> + $(foreach target,$(emulation_targets), \ > >> + $(foreach suffix,$(intermediate_suffixes), \ > >> + Build/$(binary).$(target)$(suffix)))) > >> + > >> +# In the pattern rules below, the stem (%, $*) stands for > >> +# "$(binary).$(target)". > >> + > >> +# Convert the raw ISO image to a qcow2 one, enabling compression, and using a > >> +# small cluster size. This allows for small binary files under git control, > >> +# hence for small binary patches. > >> +$(images_dir)/%.iso.qcow2: Build/%.iso.raw > >> + mkdir -p -- $(images_dir) > >> + $${QTEST_QEMU_IMG:-qemu-img} convert -f raw -O qcow2 -c \ > >> + -o cluster_size=512 -- $< $@ > >> + > >> +# Embed the "UEFI system partition" into an ISO9660 file system as an ElTorito > >> +# boot image. > >> +Build/%.iso.raw: Build/%.fat > >> + genisoimage -input-charset ASCII -efi-boot $(notdir $<) -no-emul-boot \ > >> + -quiet -o $@ -- $< > >> + > >> +# Define chained macros in order to map QEMU system emulation targets to > >> +# *short* UEFI architecture identifiers. Periods are allowed in, and ultimately > >> +# stripped from, the argument. > >> +map_arm_to_uefi = $(subst arm,ARM,$(1)) > >> +map_aarch64_to_uefi = $(subst aarch64,AA64,$(call map_arm_to_uefi,$(1))) > >> +map_i386_to_uefi = $(subst i386,IA32,$(call map_aarch64_to_uefi,$(1))) > >> +map_x86_64_to_uefi = $(subst x86_64,X64,$(call map_i386_to_uefi,$(1))) > >> +map_to_uefi = $(subst .,,$(call map_x86_64_to_uefi,$(1))) > >> + > >> +# Format a "UEFI system partition", using the UEFI binary as the default boot > >> +# loader. Add 10% size for filesystem metadata, round up to the next KB, and > >> +# make sure the size is large enough for a FAT filesystem. Name the filesystem > >> +# after the UEFI binary. (Excess characters are automatically dropped from the > >> +# filesystem label.) > >> +Build/%.fat: Build/%.efi > >> + rm -f -- $@ > >> + uefi_bin_b=$$(stat --format=%s -- $<) && \ > >> + uefi_fat_kb=$$(( (uefi_bin_b * 11 / 10 + 1023) / 1024 )) && \ > >> + uefi_fat_kb=$$(( uefi_fat_kb >= 64 ? uefi_fat_kb : 64 )) && \ > >> + mkdosfs -C $@ -n $(basename $(@F)) -- $$uefi_fat_kb > >> + MTOOLS_SKIP_CHECK=1 mmd -i $@ ::EFI > >> + MTOOLS_SKIP_CHECK=1 mmd -i $@ ::EFI/BOOT > >> + MTOOLS_SKIP_CHECK=1 mcopy -i $@ -- $< \ > >> + ::EFI/BOOT/BOOT$(call map_to_uefi,$(suffix $*)).EFI > >> + > >> +# In the pattern rules below, the stem (%, $*) stands for "$(target)" only. The > >> +# association between the UEFI binary (such as "bios-tables-test") and the > >> +# component name from the edk2 platform DSC file (such as "BiosTablesTest") is > >> +# explicit in each rule. > >> + > >> +# "build.sh" invokes the "build" utility of edk2 BaseTools. In any given edk2 > >> +# workspace, at most one "build" instance may be operating at a time. Therefore > >> +# we must serialize the rebuilding of targets in this Makefile. > >> +.NOTPARALLEL: > >> + > >> +# In turn, the "build" utility of edk2 BaseTools invokes another "make". > >> +# Although the outer "make" process advertizes its job server to all child > >> +# processes via MAKEFLAGS in the environment, the outer "make" closes the job > >> +# server file descriptors (exposed in MAKEFLAGS) before executing a recipe -- > >> +# unless the recipe is recognized as a recursive "make" recipe. Recipes that > >> +# call $(MAKE) are classified automatically as recursive; for "build.sh" below, > >> +# we must mark the recipe manually as recursive, by using the "+" indicator. > >> +# This way, when the inner "make" starts a parallel build of the target edk2 > >> +# module, it can communicate with the outer "make"'s job server. > >> +Build/bios-tables-test.%.efi: build-edk2-tools > >> + +./build.sh $(edk2_dir) BiosTablesTest $* $@ > > > > Does this actually work with an out of tree build? > > It's not supposed to. > > Again, it's not something that a normal QEMU build includes. It is only > for maintainers to rebuild when there is a reason to do so. The output > binaries are tracked by git, and will be used as-is (in binary form) by > the ACPI test suite. If there are updates to the UEFI source code, the > binaries will have to be rebuilt by a maintainer (or by me, if I submit > the UEFI code changes), and the refreshed blobs are to be checked into > git. Think iPXE oproms for an analogy. > > > Shouldn't this be SRC_PATH/tests/uefi-test-tools/ ? > > No; nothing under roms/ is built like that, and the same applies to this > patch as well. > > *Conceptually*, this patch is for roms/. However, in earlier discussion, > it was suggested that roms/ be kept dedicated to external git submodules > only, and that we not add such ROM source to roms/ whose master repo is > genuinely the QEMU repo. Please see the sub-thread at: > > Re: [PATCH 10/14] tests: acpi: ignore SMBIOS tests when UEFI firmware is used > http://mid.mail-archive.com/20190116115217.jduhqrwbjhuibmoq@sirius.home.kraxel.org > > The last idea was that the UEFI source code should be kept in a direct > subdirectory of tests/ (rather than in roms/). And the binaries should > go under tests/data/uefi-boot-images/ (rather than pc-bios/). > > Thanks > Laszlo Hmm I see. You see rebuild-expected-aml.sh does not work like this at all. It works fine with an out of tree build: check it out. So I try to keep it all out of tree. And question would be what if someone wanted a reproducible build of QEMU, what would be the right way to do it? Yes right now roms seems to be broken for an out of tree build but is that by design and should we add more examples of this?
Hi Michael, On 2/4/19 8:32 PM, Michael S. Tsirkin wrote: > On Mon, Feb 04, 2019 at 07:41:38PM +0100, Laszlo Ersek wrote: >> On 02/04/19 18:47, Michael S. Tsirkin wrote: >>> On Mon, Feb 04, 2019 at 05:03:24PM +0100, Laszlo Ersek wrote: >>>> Introduce the following build scripts under "tests/uefi-test-tools": >>>> >>>> * "build.sh" builds a single module (a UEFI application) from >>>> UefiTestToolsPkg, for a single QEMU emulation target. >>>> >>>> "build.sh" relies on cross-compilers when the emulation target and the >>>> build host architecture don't match. The cross-compiler prefix is >>>> computed according to a fixed, Linux-specific pattern. No attempt is >>>> made to copy or reimplement the GNU Make magic from "qemu/roms/Makefile" >>>> for cross-compiler prefix determination. The reason is that the build >>>> host OSes that are officially supported by edk2, and those that are >>>> supported by QEMU, intersect only in Linux. (Note that the UNIXGCC >>>> toolchain is being removed from edk2, >>>> <https://bugzilla.tianocore.org/show_bug.cgi?id=1377>.) >>>> >>>> * "Makefile" currently builds the "UefiTestToolsPkg/BiosTablesTest" >>>> application, for arm, aarch64, i386, and x86_64, with the help of >>>> "build.sh". >>>> >>>> "Makefile" turns each resultant UEFI executable into a UEFI-bootable, >>>> qcow2-compressed ISO image. The ISO images are output as >>>> "tests/data/uefi-boot-images/bios-tables-test.<TARGET>.iso.qcow2". >>>> >>>> Each ISO image should be passed to QEMU as follows: >>>> >>>> -drive id=boot-cd,if=none,readonly,format=qcow2,file=$ISO \ >>>> -device virtio-scsi-pci,id=scsi0 \ >>>> -device scsi-cd,drive=boot-cd,bus=scsi0.0,bootindex=0 \ >>>> >>>> "Makefile" assumes that "mkdosfs", "mtools", and "genisoimage" are >>>> present. >>>> >>>> Cc: "Michael S. Tsirkin" <mst@redhat.com> >>>> Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org> >>>> Cc: Gerd Hoffmann <kraxel@redhat.com> >>>> Cc: Igor Mammedov <imammedo@redhat.com> >>>> Cc: Philippe Mathieu-Daudé <philmd@redhat.com> >>>> Cc: Shannon Zhao <shannon.zhaosl@gmail.com> >>>> Signed-off-by: Laszlo Ersek <lersek@redhat.com> >>>> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> >>>> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com> >>>> --- >>>> >>>> Notes: >>>> v3: >>>> - explicitly mark the "./build.sh" recipe as recursive, with the "+" >>>> indicator; document it in a comment [Phil] >>>> - pick up R-b, T-b [Phil] >>>> >>>> v2: >>>> - add the .NOTPARALLEL target [Phil, help-make, edk2-devel] >>>> >>>> tests/uefi-test-tools/Makefile | 106 ++++++++++++++ >>>> tests/uefi-test-tools/.gitignore | 3 + >>>> tests/uefi-test-tools/build.sh | 145 ++++++++++++++++++++ >>>> 3 files changed, 254 insertions(+) >>>> >>>> diff --git a/tests/uefi-test-tools/Makefile b/tests/uefi-test-tools/Makefile >>>> new file mode 100644 >>>> index 000000000000..1d78bc14d51a >>>> --- /dev/null >>>> +++ b/tests/uefi-test-tools/Makefile >>>> @@ -0,0 +1,106 @@ >>>> +# Makefile for the test helper UEFI applications that run in guests. >>>> +# >>>> +# Copyright (C) 2019, Red Hat, Inc. >>>> +# >>>> +# This program and the accompanying materials are licensed and made available >>>> +# under the terms and conditions of the BSD License that accompanies this >>>> +# distribution. The full text of the license may be found at >>>> +# <http://opensource.org/licenses/bsd-license.php>. >>>> +# >>>> +# THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS, WITHOUT >>>> +# WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED. >>>> + >>>> +edk2_dir := ../../roms/edk2 >>>> +images_dir := ../data/uefi-boot-images >>>> +emulation_targets := arm aarch64 i386 x86_64 >>>> +uefi_binaries := bios-tables-test >>>> +intermediate_suffixes := .efi .fat .iso.raw >>>> + >>>> +images: $(foreach binary,$(uefi_binaries), \ >>>> + $(foreach target,$(emulation_targets), \ >>>> + $(images_dir)/$(binary).$(target).iso.qcow2)) >>>> + >>>> +# Preserve all intermediate targets if the build succeeds. >>>> +# - Intermediate targets help with development & debugging. >>>> +# - Preserving intermediate targets also keeps spurious changes out of the >>>> +# final build products, in case the user re-runs "make" without any changes >>>> +# to the UEFI source code. Normally, the intermediate files would have been >>>> +# removed by the last "make" invocation, hence the re-run would rebuild them >>>> +# from the unchanged UEFI sources. Unfortunately, the "mkdosfs" and >>>> +# "genisoimage" utilities embed timestamp-based information in their outputs, >>>> +# which causes git to report differences for the tracked qcow2 ISO images. >>>> +.SECONDARY: $(foreach binary,$(uefi_binaries), \ >>>> + $(foreach target,$(emulation_targets), \ >>>> + $(foreach suffix,$(intermediate_suffixes), \ >>>> + Build/$(binary).$(target)$(suffix)))) >>>> + >>>> +# In the pattern rules below, the stem (%, $*) stands for >>>> +# "$(binary).$(target)". >>>> + >>>> +# Convert the raw ISO image to a qcow2 one, enabling compression, and using a >>>> +# small cluster size. This allows for small binary files under git control, >>>> +# hence for small binary patches. >>>> +$(images_dir)/%.iso.qcow2: Build/%.iso.raw >>>> + mkdir -p -- $(images_dir) >>>> + $${QTEST_QEMU_IMG:-qemu-img} convert -f raw -O qcow2 -c \ >>>> + -o cluster_size=512 -- $< $@ >>>> + >>>> +# Embed the "UEFI system partition" into an ISO9660 file system as an ElTorito >>>> +# boot image. >>>> +Build/%.iso.raw: Build/%.fat >>>> + genisoimage -input-charset ASCII -efi-boot $(notdir $<) -no-emul-boot \ >>>> + -quiet -o $@ -- $< >>>> + >>>> +# Define chained macros in order to map QEMU system emulation targets to >>>> +# *short* UEFI architecture identifiers. Periods are allowed in, and ultimately >>>> +# stripped from, the argument. >>>> +map_arm_to_uefi = $(subst arm,ARM,$(1)) >>>> +map_aarch64_to_uefi = $(subst aarch64,AA64,$(call map_arm_to_uefi,$(1))) >>>> +map_i386_to_uefi = $(subst i386,IA32,$(call map_aarch64_to_uefi,$(1))) >>>> +map_x86_64_to_uefi = $(subst x86_64,X64,$(call map_i386_to_uefi,$(1))) >>>> +map_to_uefi = $(subst .,,$(call map_x86_64_to_uefi,$(1))) >>>> + >>>> +# Format a "UEFI system partition", using the UEFI binary as the default boot >>>> +# loader. Add 10% size for filesystem metadata, round up to the next KB, and >>>> +# make sure the size is large enough for a FAT filesystem. Name the filesystem >>>> +# after the UEFI binary. (Excess characters are automatically dropped from the >>>> +# filesystem label.) >>>> +Build/%.fat: Build/%.efi >>>> + rm -f -- $@ >>>> + uefi_bin_b=$$(stat --format=%s -- $<) && \ >>>> + uefi_fat_kb=$$(( (uefi_bin_b * 11 / 10 + 1023) / 1024 )) && \ >>>> + uefi_fat_kb=$$(( uefi_fat_kb >= 64 ? uefi_fat_kb : 64 )) && \ >>>> + mkdosfs -C $@ -n $(basename $(@F)) -- $$uefi_fat_kb >>>> + MTOOLS_SKIP_CHECK=1 mmd -i $@ ::EFI >>>> + MTOOLS_SKIP_CHECK=1 mmd -i $@ ::EFI/BOOT >>>> + MTOOLS_SKIP_CHECK=1 mcopy -i $@ -- $< \ >>>> + ::EFI/BOOT/BOOT$(call map_to_uefi,$(suffix $*)).EFI >>>> + >>>> +# In the pattern rules below, the stem (%, $*) stands for "$(target)" only. The >>>> +# association between the UEFI binary (such as "bios-tables-test") and the >>>> +# component name from the edk2 platform DSC file (such as "BiosTablesTest") is >>>> +# explicit in each rule. >>>> + >>>> +# "build.sh" invokes the "build" utility of edk2 BaseTools. In any given edk2 >>>> +# workspace, at most one "build" instance may be operating at a time. Therefore >>>> +# we must serialize the rebuilding of targets in this Makefile. >>>> +.NOTPARALLEL: >>>> + >>>> +# In turn, the "build" utility of edk2 BaseTools invokes another "make". >>>> +# Although the outer "make" process advertizes its job server to all child >>>> +# processes via MAKEFLAGS in the environment, the outer "make" closes the job >>>> +# server file descriptors (exposed in MAKEFLAGS) before executing a recipe -- >>>> +# unless the recipe is recognized as a recursive "make" recipe. Recipes that >>>> +# call $(MAKE) are classified automatically as recursive; for "build.sh" below, >>>> +# we must mark the recipe manually as recursive, by using the "+" indicator. >>>> +# This way, when the inner "make" starts a parallel build of the target edk2 >>>> +# module, it can communicate with the outer "make"'s job server. >>>> +Build/bios-tables-test.%.efi: build-edk2-tools >>>> + +./build.sh $(edk2_dir) BiosTablesTest $* $@ >>> >>> Does this actually work with an out of tree build? >> >> It's not supposed to. >> >> Again, it's not something that a normal QEMU build includes. It is only >> for maintainers to rebuild when there is a reason to do so. The output >> binaries are tracked by git, and will be used as-is (in binary form) by >> the ACPI test suite. If there are updates to the UEFI source code, the >> binaries will have to be rebuilt by a maintainer (or by me, if I submit >> the UEFI code changes), and the refreshed blobs are to be checked into >> git. Think iPXE oproms for an analogy. >> >>> Shouldn't this be SRC_PATH/tests/uefi-test-tools/ ? >> >> No; nothing under roms/ is built like that, and the same applies to this >> patch as well. >> >> *Conceptually*, this patch is for roms/. However, in earlier discussion, >> it was suggested that roms/ be kept dedicated to external git submodules >> only, and that we not add such ROM source to roms/ whose master repo is >> genuinely the QEMU repo. Please see the sub-thread at: >> >> Re: [PATCH 10/14] tests: acpi: ignore SMBIOS tests when UEFI firmware is used >> http://mid.mail-archive.com/20190116115217.jduhqrwbjhuibmoq@sirius.home.kraxel.org >> >> The last idea was that the UEFI source code should be kept in a direct >> subdirectory of tests/ (rather than in roms/). And the binaries should >> go under tests/data/uefi-boot-images/ (rather than pc-bios/). >> >> Thanks >> Laszlo > > Hmm I see. You see rebuild-expected-aml.sh does not work > like this at all. It works fine with an out of tree build: > check it out. But rebuild-expected-aml.sh does not depend of roms/. > > So I try to keep it all out of tree. Patch 5/5 add the bios-tables-test blobs under tests/data/ (as other submodules in roms/ do, adding blobs in pc-bios/). > > And question would be what if someone wanted a reproducible > build of QEMU, what would be the right way to do it? This question deserves his own thread :) > Yes right now roms seems to be broken for an out of > tree build but is that by design and should we > add more examples of this? IMO having these tests build out-of-tree is easier than trying to build various of the projects in roms/ out-of-tree. This would be a good effort, but I'm not sure it is worth it with this series. Eventually once we have a qtest using the bios-tables, we could spend some time to make this script work out-of-tree. Regards, Phil.
On Mon, Feb 04, 2019 at 08:46:33PM +0100, Philippe Mathieu-Daudé wrote: > Hi Michael, > > On 2/4/19 8:32 PM, Michael S. Tsirkin wrote: > > On Mon, Feb 04, 2019 at 07:41:38PM +0100, Laszlo Ersek wrote: > >> On 02/04/19 18:47, Michael S. Tsirkin wrote: > >>> On Mon, Feb 04, 2019 at 05:03:24PM +0100, Laszlo Ersek wrote: > >>>> Introduce the following build scripts under "tests/uefi-test-tools": > >>>> > >>>> * "build.sh" builds a single module (a UEFI application) from > >>>> UefiTestToolsPkg, for a single QEMU emulation target. > >>>> > >>>> "build.sh" relies on cross-compilers when the emulation target and the > >>>> build host architecture don't match. The cross-compiler prefix is > >>>> computed according to a fixed, Linux-specific pattern. No attempt is > >>>> made to copy or reimplement the GNU Make magic from "qemu/roms/Makefile" > >>>> for cross-compiler prefix determination. The reason is that the build > >>>> host OSes that are officially supported by edk2, and those that are > >>>> supported by QEMU, intersect only in Linux. (Note that the UNIXGCC > >>>> toolchain is being removed from edk2, > >>>> <https://bugzilla.tianocore.org/show_bug.cgi?id=1377>.) > >>>> > >>>> * "Makefile" currently builds the "UefiTestToolsPkg/BiosTablesTest" > >>>> application, for arm, aarch64, i386, and x86_64, with the help of > >>>> "build.sh". > >>>> > >>>> "Makefile" turns each resultant UEFI executable into a UEFI-bootable, > >>>> qcow2-compressed ISO image. The ISO images are output as > >>>> "tests/data/uefi-boot-images/bios-tables-test.<TARGET>.iso.qcow2". > >>>> > >>>> Each ISO image should be passed to QEMU as follows: > >>>> > >>>> -drive id=boot-cd,if=none,readonly,format=qcow2,file=$ISO \ > >>>> -device virtio-scsi-pci,id=scsi0 \ > >>>> -device scsi-cd,drive=boot-cd,bus=scsi0.0,bootindex=0 \ > >>>> > >>>> "Makefile" assumes that "mkdosfs", "mtools", and "genisoimage" are > >>>> present. > >>>> > >>>> Cc: "Michael S. Tsirkin" <mst@redhat.com> > >>>> Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org> > >>>> Cc: Gerd Hoffmann <kraxel@redhat.com> > >>>> Cc: Igor Mammedov <imammedo@redhat.com> > >>>> Cc: Philippe Mathieu-Daudé <philmd@redhat.com> > >>>> Cc: Shannon Zhao <shannon.zhaosl@gmail.com> > >>>> Signed-off-by: Laszlo Ersek <lersek@redhat.com> > >>>> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> > >>>> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com> > >>>> --- > >>>> > >>>> Notes: > >>>> v3: > >>>> - explicitly mark the "./build.sh" recipe as recursive, with the "+" > >>>> indicator; document it in a comment [Phil] > >>>> - pick up R-b, T-b [Phil] > >>>> > >>>> v2: > >>>> - add the .NOTPARALLEL target [Phil, help-make, edk2-devel] > >>>> > >>>> tests/uefi-test-tools/Makefile | 106 ++++++++++++++ > >>>> tests/uefi-test-tools/.gitignore | 3 + > >>>> tests/uefi-test-tools/build.sh | 145 ++++++++++++++++++++ > >>>> 3 files changed, 254 insertions(+) > >>>> > >>>> diff --git a/tests/uefi-test-tools/Makefile b/tests/uefi-test-tools/Makefile > >>>> new file mode 100644 > >>>> index 000000000000..1d78bc14d51a > >>>> --- /dev/null > >>>> +++ b/tests/uefi-test-tools/Makefile > >>>> @@ -0,0 +1,106 @@ > >>>> +# Makefile for the test helper UEFI applications that run in guests. > >>>> +# > >>>> +# Copyright (C) 2019, Red Hat, Inc. > >>>> +# > >>>> +# This program and the accompanying materials are licensed and made available > >>>> +# under the terms and conditions of the BSD License that accompanies this > >>>> +# distribution. The full text of the license may be found at > >>>> +# <http://opensource.org/licenses/bsd-license.php>. > >>>> +# > >>>> +# THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS, WITHOUT > >>>> +# WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED. > >>>> + > >>>> +edk2_dir := ../../roms/edk2 > >>>> +images_dir := ../data/uefi-boot-images > >>>> +emulation_targets := arm aarch64 i386 x86_64 > >>>> +uefi_binaries := bios-tables-test > >>>> +intermediate_suffixes := .efi .fat .iso.raw > >>>> + > >>>> +images: $(foreach binary,$(uefi_binaries), \ > >>>> + $(foreach target,$(emulation_targets), \ > >>>> + $(images_dir)/$(binary).$(target).iso.qcow2)) > >>>> + > >>>> +# Preserve all intermediate targets if the build succeeds. > >>>> +# - Intermediate targets help with development & debugging. > >>>> +# - Preserving intermediate targets also keeps spurious changes out of the > >>>> +# final build products, in case the user re-runs "make" without any changes > >>>> +# to the UEFI source code. Normally, the intermediate files would have been > >>>> +# removed by the last "make" invocation, hence the re-run would rebuild them > >>>> +# from the unchanged UEFI sources. Unfortunately, the "mkdosfs" and > >>>> +# "genisoimage" utilities embed timestamp-based information in their outputs, > >>>> +# which causes git to report differences for the tracked qcow2 ISO images. > >>>> +.SECONDARY: $(foreach binary,$(uefi_binaries), \ > >>>> + $(foreach target,$(emulation_targets), \ > >>>> + $(foreach suffix,$(intermediate_suffixes), \ > >>>> + Build/$(binary).$(target)$(suffix)))) > >>>> + > >>>> +# In the pattern rules below, the stem (%, $*) stands for > >>>> +# "$(binary).$(target)". > >>>> + > >>>> +# Convert the raw ISO image to a qcow2 one, enabling compression, and using a > >>>> +# small cluster size. This allows for small binary files under git control, > >>>> +# hence for small binary patches. > >>>> +$(images_dir)/%.iso.qcow2: Build/%.iso.raw > >>>> + mkdir -p -- $(images_dir) > >>>> + $${QTEST_QEMU_IMG:-qemu-img} convert -f raw -O qcow2 -c \ > >>>> + -o cluster_size=512 -- $< $@ > >>>> + > >>>> +# Embed the "UEFI system partition" into an ISO9660 file system as an ElTorito > >>>> +# boot image. > >>>> +Build/%.iso.raw: Build/%.fat > >>>> + genisoimage -input-charset ASCII -efi-boot $(notdir $<) -no-emul-boot \ > >>>> + -quiet -o $@ -- $< > >>>> + > >>>> +# Define chained macros in order to map QEMU system emulation targets to > >>>> +# *short* UEFI architecture identifiers. Periods are allowed in, and ultimately > >>>> +# stripped from, the argument. > >>>> +map_arm_to_uefi = $(subst arm,ARM,$(1)) > >>>> +map_aarch64_to_uefi = $(subst aarch64,AA64,$(call map_arm_to_uefi,$(1))) > >>>> +map_i386_to_uefi = $(subst i386,IA32,$(call map_aarch64_to_uefi,$(1))) > >>>> +map_x86_64_to_uefi = $(subst x86_64,X64,$(call map_i386_to_uefi,$(1))) > >>>> +map_to_uefi = $(subst .,,$(call map_x86_64_to_uefi,$(1))) > >>>> + > >>>> +# Format a "UEFI system partition", using the UEFI binary as the default boot > >>>> +# loader. Add 10% size for filesystem metadata, round up to the next KB, and > >>>> +# make sure the size is large enough for a FAT filesystem. Name the filesystem > >>>> +# after the UEFI binary. (Excess characters are automatically dropped from the > >>>> +# filesystem label.) > >>>> +Build/%.fat: Build/%.efi > >>>> + rm -f -- $@ > >>>> + uefi_bin_b=$$(stat --format=%s -- $<) && \ > >>>> + uefi_fat_kb=$$(( (uefi_bin_b * 11 / 10 + 1023) / 1024 )) && \ > >>>> + uefi_fat_kb=$$(( uefi_fat_kb >= 64 ? uefi_fat_kb : 64 )) && \ > >>>> + mkdosfs -C $@ -n $(basename $(@F)) -- $$uefi_fat_kb > >>>> + MTOOLS_SKIP_CHECK=1 mmd -i $@ ::EFI > >>>> + MTOOLS_SKIP_CHECK=1 mmd -i $@ ::EFI/BOOT > >>>> + MTOOLS_SKIP_CHECK=1 mcopy -i $@ -- $< \ > >>>> + ::EFI/BOOT/BOOT$(call map_to_uefi,$(suffix $*)).EFI > >>>> + > >>>> +# In the pattern rules below, the stem (%, $*) stands for "$(target)" only. The > >>>> +# association between the UEFI binary (such as "bios-tables-test") and the > >>>> +# component name from the edk2 platform DSC file (such as "BiosTablesTest") is > >>>> +# explicit in each rule. > >>>> + > >>>> +# "build.sh" invokes the "build" utility of edk2 BaseTools. In any given edk2 > >>>> +# workspace, at most one "build" instance may be operating at a time. Therefore > >>>> +# we must serialize the rebuilding of targets in this Makefile. > >>>> +.NOTPARALLEL: > >>>> + > >>>> +# In turn, the "build" utility of edk2 BaseTools invokes another "make". > >>>> +# Although the outer "make" process advertizes its job server to all child > >>>> +# processes via MAKEFLAGS in the environment, the outer "make" closes the job > >>>> +# server file descriptors (exposed in MAKEFLAGS) before executing a recipe -- > >>>> +# unless the recipe is recognized as a recursive "make" recipe. Recipes that > >>>> +# call $(MAKE) are classified automatically as recursive; for "build.sh" below, > >>>> +# we must mark the recipe manually as recursive, by using the "+" indicator. > >>>> +# This way, when the inner "make" starts a parallel build of the target edk2 > >>>> +# module, it can communicate with the outer "make"'s job server. > >>>> +Build/bios-tables-test.%.efi: build-edk2-tools > >>>> + +./build.sh $(edk2_dir) BiosTablesTest $* $@ > >>> > >>> Does this actually work with an out of tree build? > >> > >> It's not supposed to. > >> > >> Again, it's not something that a normal QEMU build includes. It is only > >> for maintainers to rebuild when there is a reason to do so. The output > >> binaries are tracked by git, and will be used as-is (in binary form) by > >> the ACPI test suite. If there are updates to the UEFI source code, the > >> binaries will have to be rebuilt by a maintainer (or by me, if I submit > >> the UEFI code changes), and the refreshed blobs are to be checked into > >> git. Think iPXE oproms for an analogy. > >> > >>> Shouldn't this be SRC_PATH/tests/uefi-test-tools/ ? > >> > >> No; nothing under roms/ is built like that, and the same applies to this > >> patch as well. > >> > >> *Conceptually*, this patch is for roms/. However, in earlier discussion, > >> it was suggested that roms/ be kept dedicated to external git submodules > >> only, and that we not add such ROM source to roms/ whose master repo is > >> genuinely the QEMU repo. Please see the sub-thread at: > >> > >> Re: [PATCH 10/14] tests: acpi: ignore SMBIOS tests when UEFI firmware is used > >> http://mid.mail-archive.com/20190116115217.jduhqrwbjhuibmoq@sirius.home.kraxel.org > >> > >> The last idea was that the UEFI source code should be kept in a direct > >> subdirectory of tests/ (rather than in roms/). And the binaries should > >> go under tests/data/uefi-boot-images/ (rather than pc-bios/). > >> > >> Thanks > >> Laszlo > > > > Hmm I see. You see rebuild-expected-aml.sh does not work > > like this at all. It works fine with an out of tree build: > > check it out. > > But rebuild-expected-aml.sh does not depend of roms/. It absolutely depends on pc-bios/bios.bin > > > > So I try to keep it all out of tree. > > Patch 5/5 add the bios-tables-test blobs under tests/data/ (as other > submodules in roms/ do, adding blobs in pc-bios/). > > > > > And question would be what if someone wanted a reproducible > > build of QEMU, what would be the right way to do it? > > This question deserves his own thread :) Sure. > > Yes right now roms seems to be broken for an out of > > tree build but is that by design and should we > > add more examples of this? > > IMO having these tests build out-of-tree is easier than trying to build > various of the projects in roms/ out-of-tree. > This would be a good effort, but I'm not sure it is worth it with this > series. > Eventually once we have a qtest using the bios-tables, we could spend > some time to make this script work out-of-tree. > > Regards, > > Phil. I'm not saying it's a blocker.
On 2/4/19 10:55 PM, Michael S. Tsirkin wrote: > On Mon, Feb 04, 2019 at 08:46:33PM +0100, Philippe Mathieu-Daudé wrote: >> On 2/4/19 8:32 PM, Michael S. Tsirkin wrote: >>> On Mon, Feb 04, 2019 at 07:41:38PM +0100, Laszlo Ersek wrote: >>>> On 02/04/19 18:47, Michael S. Tsirkin wrote: >>>>> On Mon, Feb 04, 2019 at 05:03:24PM +0100, Laszlo Ersek wrote: >>>>>> Introduce the following build scripts under "tests/uefi-test-tools": [...]>>>>>> +# In turn, the "build" utility of edk2 BaseTools invokes another "make". >>>>>> +# Although the outer "make" process advertizes its job server to all child >>>>>> +# processes via MAKEFLAGS in the environment, the outer "make" closes the job >>>>>> +# server file descriptors (exposed in MAKEFLAGS) before executing a recipe -- >>>>>> +# unless the recipe is recognized as a recursive "make" recipe. Recipes that >>>>>> +# call $(MAKE) are classified automatically as recursive; for "build.sh" below, >>>>>> +# we must mark the recipe manually as recursive, by using the "+" indicator. >>>>>> +# This way, when the inner "make" starts a parallel build of the target edk2 >>>>>> +# module, it can communicate with the outer "make"'s job server. >>>>>> +Build/bios-tables-test.%.efi: build-edk2-tools >>>>>> + +./build.sh $(edk2_dir) BiosTablesTest $* $@ >>>>> >>>>> Does this actually work with an out of tree build? >>>> >>>> It's not supposed to. >>>> >>>> Again, it's not something that a normal QEMU build includes. It is only >>>> for maintainers to rebuild when there is a reason to do so. The output >>>> binaries are tracked by git, and will be used as-is (in binary form) by >>>> the ACPI test suite. If there are updates to the UEFI source code, the >>>> binaries will have to be rebuilt by a maintainer (or by me, if I submit >>>> the UEFI code changes), and the refreshed blobs are to be checked into >>>> git. Think iPXE oproms for an analogy. >>>> >>>>> Shouldn't this be SRC_PATH/tests/uefi-test-tools/ ? >>>> >>>> No; nothing under roms/ is built like that, and the same applies to this >>>> patch as well. >>>> >>>> *Conceptually*, this patch is for roms/. However, in earlier discussion, >>>> it was suggested that roms/ be kept dedicated to external git submodules >>>> only, and that we not add such ROM source to roms/ whose master repo is >>>> genuinely the QEMU repo. Please see the sub-thread at: >>>> >>>> Re: [PATCH 10/14] tests: acpi: ignore SMBIOS tests when UEFI firmware is used >>>> http://mid.mail-archive.com/20190116115217.jduhqrwbjhuibmoq@sirius.home.kraxel.org >>>> >>>> The last idea was that the UEFI source code should be kept in a direct >>>> subdirectory of tests/ (rather than in roms/). And the binaries should >>>> go under tests/data/uefi-boot-images/ (rather than pc-bios/). >>>> >>>> Thanks >>>> Laszlo >>> >>> Hmm I see. You see rebuild-expected-aml.sh does not work >>> like this at all. It works fine with an out of tree build: >>> check it out. >> >> But rebuild-expected-aml.sh does not depend of roms/. > > It absolutely depends on pc-bios/bios.bin Oh you are right! I'm taking notes for an eventual future cleanup. Regards, Phil.
On 02/04/19 22:55, Michael S. Tsirkin wrote: > On Mon, Feb 04, 2019 at 08:46:33PM +0100, Philippe Mathieu-Daudé > wrote: >> Hi Michael, >> >> On 2/4/19 8:32 PM, Michael S. Tsirkin wrote: >>> And question would be what if someone wanted a reproducible >>> build of QEMU, what would be the right way to do it? >> >> This question deserves his own thread :) > > Sure. (I don't know enough to comment on this sensibly; so I'm only following up to show that I'm not willfully ignoring the topic.) What constitutes a "QEMU build" in this context? If it means the sum of artifacts that get installed by "make install", then the current patch set doesn't factor into that at all. If it means something else, then "it depends". For example, "genisoimage" and "mkdosfs" place pseudo-random / timestamp-based identifiers into the images that they generate, and there's no way to prevent at least "genisoimage" from doing that. Thus, if we consider UEFI-bootable ISO images, rebuilt from zero, a part of a "reproducible build", then the answer is "there's no right way until someone extends genisoimage with some new cmdline options". >>> Yes right now roms seems to be broken for an out of tree build but >>> is that by design and should we add more examples of this? >> >> IMO having these tests build out-of-tree is easier than trying to >> build various of the projects in roms/ out-of-tree. This would be a >> good effort, but I'm not sure it is worth it with this series. >> Eventually once we have a qtest using the bios-tables, we could spend >> some time to make this script work out-of-tree. > > I'm not saying it's a blocker. Regarding roms/edk2, we have to distinguish two sets of source code, wrt. building out-of-tree. - The firmware modules build just fine outside of the tree, and this patch set already puts that feature to use. (Namely, all the dependencies that the tests/uefi-test-tools/UefiTestToolsPkg/BiosTablesTest/BiosTablesTest.inf application pulls in from roms/edk2 are compiled outside of roms/edk2.) - However, the native build utilities (in the roms/edk2/BaseTools subdirectory) that first have to be built natively, so that they can help produce the firmware executables from the firmware source, don't themselves build outside of that subdirectory. This is a genuine limitation of upstream edk2, as far as I can tell. Thanks, Laszlo
On Tue, Feb 05, 2019 at 09:49:57AM +0100, Laszlo Ersek wrote: > On 02/04/19 22:55, Michael S. Tsirkin wrote: > > On Mon, Feb 04, 2019 at 08:46:33PM +0100, Philippe Mathieu-Daudé > > wrote: > >> Hi Michael, > >> > >> On 2/4/19 8:32 PM, Michael S. Tsirkin wrote: > > >>> And question would be what if someone wanted a reproducible > >>> build of QEMU, what would be the right way to do it? > >> > >> This question deserves his own thread :) > > > > Sure. > > (I don't know enough to comment on this sensibly; so I'm only following > up to show that I'm not willfully ignoring the topic.) Thanks! > What constitutes a "QEMU build" in this context? > > If it means the sum of artifacts that get installed by "make install", > then the current patch set doesn't factor into that at all. That's true in that it's just a test. > If it means something else, then "it depends". For example, > "genisoimage" and "mkdosfs" place pseudo-random / timestamp-based > identifiers into the images that they generate, and there's no way to > prevent at least "genisoimage" from doing that. Thus, if we consider > UEFI-bootable ISO images, rebuilt from zero, a part of a "reproducible > build", then the answer is "there's no right way until someone extends > genisoimage with some new cmdline options". Yep that's true too. > >>> Yes right now roms seems to be broken for an out of tree build but > >>> is that by design and should we add more examples of this? > >> > >> IMO having these tests build out-of-tree is easier than trying to > >> build various of the projects in roms/ out-of-tree. This would be a > >> good effort, but I'm not sure it is worth it with this series. > >> Eventually once we have a qtest using the bios-tables, we could spend > >> some time to make this script work out-of-tree. > > > > I'm not saying it's a blocker. > > Regarding roms/edk2, we have to distinguish two sets of source code, > wrt. building out-of-tree. > > - The firmware modules build just fine outside of the tree, and this > patch set already puts that feature to use. (Namely, all the > dependencies that the > > tests/uefi-test-tools/UefiTestToolsPkg/BiosTablesTest/BiosTablesTest.inf > > application pulls in from roms/edk2 are compiled outside of > roms/edk2.) > > - However, the native build utilities (in the > > roms/edk2/BaseTools > > subdirectory) that first have to be built natively, so that they can > help produce the firmware executables from the firmware source, don't > themselves build outside of that subdirectory. This is a genuine > limitation of upstream edk2, as far as I can tell. > > Thanks, > Laszlo Interesting, thanks for sharing.
diff --git a/tests/uefi-test-tools/Makefile b/tests/uefi-test-tools/Makefile new file mode 100644 index 000000000000..1d78bc14d51a --- /dev/null +++ b/tests/uefi-test-tools/Makefile @@ -0,0 +1,106 @@ +# Makefile for the test helper UEFI applications that run in guests. +# +# Copyright (C) 2019, Red Hat, Inc. +# +# This program and the accompanying materials are licensed and made available +# under the terms and conditions of the BSD License that accompanies this +# distribution. The full text of the license may be found at +# <http://opensource.org/licenses/bsd-license.php>. +# +# THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS, WITHOUT +# WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED. + +edk2_dir := ../../roms/edk2 +images_dir := ../data/uefi-boot-images +emulation_targets := arm aarch64 i386 x86_64 +uefi_binaries := bios-tables-test +intermediate_suffixes := .efi .fat .iso.raw + +images: $(foreach binary,$(uefi_binaries), \ + $(foreach target,$(emulation_targets), \ + $(images_dir)/$(binary).$(target).iso.qcow2)) + +# Preserve all intermediate targets if the build succeeds. +# - Intermediate targets help with development & debugging. +# - Preserving intermediate targets also keeps spurious changes out of the +# final build products, in case the user re-runs "make" without any changes +# to the UEFI source code. Normally, the intermediate files would have been +# removed by the last "make" invocation, hence the re-run would rebuild them +# from the unchanged UEFI sources. Unfortunately, the "mkdosfs" and +# "genisoimage" utilities embed timestamp-based information in their outputs, +# which causes git to report differences for the tracked qcow2 ISO images. +.SECONDARY: $(foreach binary,$(uefi_binaries), \ + $(foreach target,$(emulation_targets), \ + $(foreach suffix,$(intermediate_suffixes), \ + Build/$(binary).$(target)$(suffix)))) + +# In the pattern rules below, the stem (%, $*) stands for +# "$(binary).$(target)". + +# Convert the raw ISO image to a qcow2 one, enabling compression, and using a +# small cluster size. This allows for small binary files under git control, +# hence for small binary patches. +$(images_dir)/%.iso.qcow2: Build/%.iso.raw + mkdir -p -- $(images_dir) + $${QTEST_QEMU_IMG:-qemu-img} convert -f raw -O qcow2 -c \ + -o cluster_size=512 -- $< $@ + +# Embed the "UEFI system partition" into an ISO9660 file system as an ElTorito +# boot image. +Build/%.iso.raw: Build/%.fat + genisoimage -input-charset ASCII -efi-boot $(notdir $<) -no-emul-boot \ + -quiet -o $@ -- $< + +# Define chained macros in order to map QEMU system emulation targets to +# *short* UEFI architecture identifiers. Periods are allowed in, and ultimately +# stripped from, the argument. +map_arm_to_uefi = $(subst arm,ARM,$(1)) +map_aarch64_to_uefi = $(subst aarch64,AA64,$(call map_arm_to_uefi,$(1))) +map_i386_to_uefi = $(subst i386,IA32,$(call map_aarch64_to_uefi,$(1))) +map_x86_64_to_uefi = $(subst x86_64,X64,$(call map_i386_to_uefi,$(1))) +map_to_uefi = $(subst .,,$(call map_x86_64_to_uefi,$(1))) + +# Format a "UEFI system partition", using the UEFI binary as the default boot +# loader. Add 10% size for filesystem metadata, round up to the next KB, and +# make sure the size is large enough for a FAT filesystem. Name the filesystem +# after the UEFI binary. (Excess characters are automatically dropped from the +# filesystem label.) +Build/%.fat: Build/%.efi + rm -f -- $@ + uefi_bin_b=$$(stat --format=%s -- $<) && \ + uefi_fat_kb=$$(( (uefi_bin_b * 11 / 10 + 1023) / 1024 )) && \ + uefi_fat_kb=$$(( uefi_fat_kb >= 64 ? uefi_fat_kb : 64 )) && \ + mkdosfs -C $@ -n $(basename $(@F)) -- $$uefi_fat_kb + MTOOLS_SKIP_CHECK=1 mmd -i $@ ::EFI + MTOOLS_SKIP_CHECK=1 mmd -i $@ ::EFI/BOOT + MTOOLS_SKIP_CHECK=1 mcopy -i $@ -- $< \ + ::EFI/BOOT/BOOT$(call map_to_uefi,$(suffix $*)).EFI + +# In the pattern rules below, the stem (%, $*) stands for "$(target)" only. The +# association between the UEFI binary (such as "bios-tables-test") and the +# component name from the edk2 platform DSC file (such as "BiosTablesTest") is +# explicit in each rule. + +# "build.sh" invokes the "build" utility of edk2 BaseTools. In any given edk2 +# workspace, at most one "build" instance may be operating at a time. Therefore +# we must serialize the rebuilding of targets in this Makefile. +.NOTPARALLEL: + +# In turn, the "build" utility of edk2 BaseTools invokes another "make". +# Although the outer "make" process advertizes its job server to all child +# processes via MAKEFLAGS in the environment, the outer "make" closes the job +# server file descriptors (exposed in MAKEFLAGS) before executing a recipe -- +# unless the recipe is recognized as a recursive "make" recipe. Recipes that +# call $(MAKE) are classified automatically as recursive; for "build.sh" below, +# we must mark the recipe manually as recursive, by using the "+" indicator. +# This way, when the inner "make" starts a parallel build of the target edk2 +# module, it can communicate with the outer "make"'s job server. +Build/bios-tables-test.%.efi: build-edk2-tools + +./build.sh $(edk2_dir) BiosTablesTest $* $@ + +build-edk2-tools: + $(MAKE) -C $(edk2_dir)/BaseTools + +clean: + rm -rf Build Conf log + $(MAKE) -C $(edk2_dir)/BaseTools clean diff --git a/tests/uefi-test-tools/.gitignore b/tests/uefi-test-tools/.gitignore new file mode 100644 index 000000000000..9f246701dea1 --- /dev/null +++ b/tests/uefi-test-tools/.gitignore @@ -0,0 +1,3 @@ +Build +Conf +log diff --git a/tests/uefi-test-tools/build.sh b/tests/uefi-test-tools/build.sh new file mode 100755 index 000000000000..155cb75c4ddb --- /dev/null +++ b/tests/uefi-test-tools/build.sh @@ -0,0 +1,145 @@ +#!/bin/bash + +# Build script that determines the edk2 toolchain to use, invokes the edk2 +# "build" utility, and copies the built UEFI binary to the requested location. +# +# Copyright (C) 2019, Red Hat, Inc. +# +# This program and the accompanying materials are licensed and made available +# under the terms and conditions of the BSD License that accompanies this +# distribution. The full text of the license may be found at +# <http://opensource.org/licenses/bsd-license.php>. +# +# THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS, WITHOUT +# WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED. + +set -e -u -C + +# Save the command line arguments. We need to reset $# to 0 before sourcing +# "edksetup.sh", as it will inherit $@. +program_name=$(basename -- "$0") +edk2_dir=$1 +dsc_component=$2 +emulation_target=$3 +uefi_binary=$4 +shift 4 + +# Set up the environment for edk2 building. +export PACKAGES_PATH=$(realpath -- "$edk2_dir") +export WORKSPACE=$PWD +mkdir -p Conf + +# Source "edksetup.sh" carefully. +set +e +u +C +source "$PACKAGES_PATH/edksetup.sh" +ret=$? +set -e -u -C +if [ $ret -ne 0 ]; then + exit $ret +fi + +# Map the QEMU system emulation target to the following types of architecture +# identifiers: +# - edk2, +# - gcc cross-compilation. +# Cover only those targets that are supported by the UEFI spec and edk2. +case "$emulation_target" in + (arm) + edk2_arch=ARM + gcc_arch=arm + ;; + (aarch64) + edk2_arch=AARCH64 + gcc_arch=aarch64 + ;; + (i386) + edk2_arch=IA32 + gcc_arch=i686 + ;; + (x86_64) + edk2_arch=X64 + gcc_arch=x86_64 + ;; + (*) + printf '%s: unknown/unsupported QEMU system emulation target "%s"\n' \ + "$program_name" "$emulation_target" >&2 + exit 1 + ;; +esac + +# Check if cross-compilation is needed. +host_arch=$(uname -m) +if [ "$gcc_arch" == "$host_arch" ] || + ( [ "$gcc_arch" == i686 ] && [ "$host_arch" == x86_64 ] ); then + cross_prefix= +else + cross_prefix=${gcc_arch}-linux-gnu- +fi + +# Expose cross_prefix (which is possibly empty) to the edk2 tools. While at it, +# determine the suitable edk2 toolchain as well. +# - For ARM and AARCH64, edk2 only offers the GCC5 toolchain tag, which covers +# the gcc-5+ releases. +# - For IA32 and X64, edk2 offers the GCC44 through GCC49 toolchain tags, in +# addition to GCC5. Unfortunately, the mapping between the toolchain tags and +# the actual gcc releases isn't entirely trivial. Run "git-blame" on +# "OvmfPkg/build.sh" in edk2 for more information. +# And, because the above is too simple, we have to assign cross_prefix to an +# edk2 build variable that is specific to both the toolchain tag and the target +# architecture. +case "$edk2_arch" in + (ARM) + edk2_toolchain=GCC5 + export GCC5_ARM_PREFIX=$cross_prefix + ;; + (AARCH64) + edk2_toolchain=GCC5 + export GCC5_AARCH64_PREFIX=$cross_prefix + ;; + (IA32|X64) + gcc_version=$("${cross_prefix}gcc" -v 2>&1 | tail -1 | awk '{print $3}') + case "$gcc_version" in + ([1-3].*|4.[0-3].*) + printf '%s: unsupported gcc version "%s"\n' \ + "$program_name" "$gcc_version" >&2 + exit 1 + ;; + (4.4.*) + edk2_toolchain=GCC44 + ;; + (4.5.*) + edk2_toolchain=GCC45 + ;; + (4.6.*) + edk2_toolchain=GCC46 + ;; + (4.7.*) + edk2_toolchain=GCC47 + ;; + (4.8.*) + edk2_toolchain=GCC48 + ;; + (4.9.*|6.[0-2].*) + edk2_toolchain=GCC49 + ;; + (*) + edk2_toolchain=GCC5 + ;; + esac + eval "export ${edk2_toolchain}_BIN=\$cross_prefix" + ;; +esac + +# Build the UEFI binary +mkdir -p log +build \ + --arch="$edk2_arch" \ + --buildtarget=DEBUG \ + --platform=UefiTestToolsPkg/UefiTestToolsPkg.dsc \ + --tagname="$edk2_toolchain" \ + --module="UefiTestToolsPkg/$dsc_component/$dsc_component.inf" \ + --log="log/$dsc_component.$edk2_arch.log" \ + --report-file="log/$dsc_component.$edk2_arch.report" +cp -a -- \ + "Build/UefiTestTools/DEBUG_${edk2_toolchain}/$edk2_arch/$dsc_component.efi" \ + "$uefi_binary"
Introduce the following build scripts under "tests/uefi-test-tools": * "build.sh" builds a single module (a UEFI application) from UefiTestToolsPkg, for a single QEMU emulation target. "build.sh" relies on cross-compilers when the emulation target and the build host architecture don't match. The cross-compiler prefix is computed according to a fixed, Linux-specific pattern. No attempt is made to copy or reimplement the GNU Make magic from "qemu/roms/Makefile" for cross-compiler prefix determination. The reason is that the build host OSes that are officially supported by edk2, and those that are supported by QEMU, intersect only in Linux. (Note that the UNIXGCC toolchain is being removed from edk2, <https://bugzilla.tianocore.org/show_bug.cgi?id=1377>.) * "Makefile" currently builds the "UefiTestToolsPkg/BiosTablesTest" application, for arm, aarch64, i386, and x86_64, with the help of "build.sh". "Makefile" turns each resultant UEFI executable into a UEFI-bootable, qcow2-compressed ISO image. The ISO images are output as "tests/data/uefi-boot-images/bios-tables-test.<TARGET>.iso.qcow2". Each ISO image should be passed to QEMU as follows: -drive id=boot-cd,if=none,readonly,format=qcow2,file=$ISO \ -device virtio-scsi-pci,id=scsi0 \ -device scsi-cd,drive=boot-cd,bus=scsi0.0,bootindex=0 \ "Makefile" assumes that "mkdosfs", "mtools", and "genisoimage" are present. Cc: "Michael S. Tsirkin" <mst@redhat.com> Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org> Cc: Gerd Hoffmann <kraxel@redhat.com> Cc: Igor Mammedov <imammedo@redhat.com> Cc: Philippe Mathieu-Daudé <philmd@redhat.com> Cc: Shannon Zhao <shannon.zhaosl@gmail.com> Signed-off-by: Laszlo Ersek <lersek@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com> --- Notes: v3: - explicitly mark the "./build.sh" recipe as recursive, with the "+" indicator; document it in a comment [Phil] - pick up R-b, T-b [Phil] v2: - add the .NOTPARALLEL target [Phil, help-make, edk2-devel] tests/uefi-test-tools/Makefile | 106 ++++++++++++++ tests/uefi-test-tools/.gitignore | 3 + tests/uefi-test-tools/build.sh | 145 ++++++++++++++++++++ 3 files changed, 254 insertions(+)