| Message ID | 20190118223400.24311-5-lersek@redhat.com |
|---|---|
| State | New |
| Headers | show |
| Series | add the BiosTablesTest UEFI app, build it with the new roms/edk2 submodule | expand |
Hi Laszlo, On 1/18/19 11:33 PM, Laszlo Ersek wrote: > Introduce the following build scripts under "tests/uefi-test-tools": > > * "build.sh" builds a single module (a UEFI application) from > UefiTestToolsPkg, for a single QEMU emulation target. > > "build.sh" relies on cross-compilers when the emulation target and the > build host architecture don't match. The cross-compiler prefix is > computed according to a fixed, Linux-specific pattern. No attempt is > made to copy or reimplement the GNU Make magic from "qemu/roms/Makefile" > for cross-compiler prefix determination. The reason is that the build > host OSes that are officially supported by edk2, and those that are > supported by QEMU, intersect only in Linux. (Note that the UNIXGCC > toolchain is being removed from edk2, > <https://bugzilla.tianocore.org/show_bug.cgi?id=1377>.) > > * "Makefile" currently builds the "UefiTestToolsPkg/BiosTablesTest" > application, for arm, aarch64, i386, and x86_64, with the help of > "build.sh". > > "Makefile" turns each resultant UEFI executable into a UEFI-bootable, > qcow2-compressed ISO image. The ISO images are output as > "tests/data/uefi-boot-images/bios-tables-test.<TARGET>.iso.qcow2". > > Each ISO image should be passed to QEMU as follows: > > -drive id=boot-cd,if=none,readonly,format=qcow2,file=$ISO \ > -device virtio-scsi-pci,id=scsi0 \ > -device scsi-cd,drive=boot-cd,bus=scsi0.0,bootindex=0 \ > > "Makefile" assumes that "mkdosfs", "mtools", and "genisoimage" are > present. > > Cc: "Michael S. Tsirkin" <mst@redhat.com> > Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org> > Cc: Gerd Hoffmann <kraxel@redhat.com> > Cc: Igor Mammedov <imammedo@redhat.com> > Cc: Philippe Mathieu-Daudé <philmd@redhat.com> > Cc: Shannon Zhao <shannon.zhaosl@gmail.com> > Signed-off-by: Laszlo Ersek <lersek@redhat.com> > --- > tests/uefi-test-tools/Makefile | 92 +++++++++++++ > tests/uefi-test-tools/.gitignore | 3 + > tests/uefi-test-tools/build.sh | 145 ++++++++++++++++++++ > 3 files changed, 240 insertions(+) > > diff --git a/tests/uefi-test-tools/Makefile b/tests/uefi-test-tools/Makefile > new file mode 100644 > index 000000000000..7b6dd227e433 > --- /dev/null > +++ b/tests/uefi-test-tools/Makefile > @@ -0,0 +1,92 @@ > +# Makefile for the test helper UEFI applications that run in guests. > +# > +# Copyright (C) 2019, Red Hat, Inc. > +# > +# This program and the accompanying materials are licensed and made available > +# under the terms and conditions of the BSD License that accompanies this > +# distribution. The full text of the license may be found at > +# <http://opensource.org/licenses/bsd-license.php>. > +# > +# THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS, WITHOUT > +# WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED. > + > +edk2_dir := ../../roms/edk2 > +images_dir := ../data/uefi-boot-images > +emulation_targets := arm aarch64 i386 x86_64 > +uefi_binaries := bios-tables-test > +intermediate_suffixes := .efi .fat .iso.raw > + > +images: $(foreach binary,$(uefi_binaries), \ > + $(foreach target,$(emulation_targets), \ > + $(images_dir)/$(binary).$(target).iso.qcow2)) > + > +# Preserve all intermediate targets if the build succeeds. > +# - Intermediate targets help with development & debugging. > +# - Preserving intermediate targets also keeps spurious changes out of the > +# final build products, in case the user re-runs "make" without any changes > +# to the UEFI source code. Normally, the intermediate files would have been > +# removed by the last "make" invocation, hence the re-run would rebuild them > +# from the unchanged UEFI sources. Unfortunately, the "mkdosfs" and > +# "genisoimage" utilities embed timestamp-based information in their outputs, > +# which causes git to report differences for the tracked qcow2 ISO images. > +.SECONDARY: $(foreach binary,$(uefi_binaries), \ > + $(foreach target,$(emulation_targets), \ > + $(foreach suffix,$(intermediate_suffixes), \ > + Build/$(binary).$(target)$(suffix)))) > + > +# In the pattern rules below, the stem (%, $*) stands for > +# "$(binary).$(target)". > + > +# Convert the raw ISO image to a qcow2 one, enabling compression, and using a > +# small cluster size. This allows for small binary files under git control, > +# hence for small binary patches. > +$(images_dir)/%.iso.qcow2: Build/%.iso.raw > + mkdir -p -- $(images_dir) > + $${QTEST_QEMU_IMG:-qemu-img} convert -f raw -O qcow2 -c \ > + -o cluster_size=512 -- $< $@ > + > +# Embed the "UEFI system partition" into an ISO9660 file system as an ElTorito > +# boot image. > +Build/%.iso.raw: Build/%.fat > + genisoimage -input-charset ASCII -efi-boot $(notdir $<) -no-emul-boot \ > + -quiet -o $@ -- $< > + > +# Define chained macros in order to map QEMU system emulation targets to > +# *short* UEFI architecture identifiers. Periods are allowed in, and ultimately > +# stripped from, the argument. > +map_arm_to_uefi = $(subst arm,ARM,$(1)) > +map_aarch64_to_uefi = $(subst aarch64,AA64,$(call map_arm_to_uefi,$(1))) > +map_i386_to_uefi = $(subst i386,IA32,$(call map_aarch64_to_uefi,$(1))) > +map_x86_64_to_uefi = $(subst x86_64,X64,$(call map_i386_to_uefi,$(1))) > +map_to_uefi = $(subst .,,$(call map_x86_64_to_uefi,$(1))) > + > +# Format a "UEFI system partition", using the UEFI binary as the default boot > +# loader. Add 10% size for filesystem metadata, round up to the next KB, and > +# make sure the size is large enough for a FAT filesystem. Name the filesystem > +# after the UEFI binary. (Excess characters are automatically dropped from the > +# filesystem label.) > +Build/%.fat: Build/%.efi > + rm -f -- $@ > + uefi_bin_b=$$(stat --format=%s -- $<) && \ > + uefi_fat_kb=$$(( (uefi_bin_b * 11 / 10 + 1023) / 1024 )) && \ > + uefi_fat_kb=$$(( uefi_fat_kb >= 64 ? uefi_fat_kb : 64 )) && \ > + mkdosfs -C $@ -n $(basename $(@F)) -- $$uefi_fat_kb > + MTOOLS_SKIP_CHECK=1 mmd -i $@ ::EFI > + MTOOLS_SKIP_CHECK=1 mmd -i $@ ::EFI/BOOT > + MTOOLS_SKIP_CHECK=1 mcopy -i $@ -- $< \ > + ::EFI/BOOT/BOOT$(call map_to_uefi,$(suffix $*)).EFI > + > +# In the pattern rules below, the stem (%, $*) stands for "$(target)" only. The > +# association between the UEFI binary (such as "bios-tables-test") and the > +# component name from the edk2 platform DSC file (such as "BiosTablesTest") is > +# explicit in each rule. > + > +Build/bios-tables-test.%.efi: build-edk2-tools > + ./build.sh $(edk2_dir) BiosTablesTest $* $@ > + > +build-edk2-tools: > + $(MAKE) -C $(edk2_dir)/BaseTools > + [...] I got errors [1] and [2] I couldn't figure out while running 'make -j4'. The following patch didn't help, any idea? -- >8 -- diff --git a/tests/uefi-test-tools/Makefile b/tests/uefi-test-tools/Makefile index 7b6dd227e4..798c55c823 100644 --- a/tests/uefi-test-tools/Makefile +++ b/tests/uefi-test-tools/Makefile @@ -81,6 +81,8 @@ Build/%.fat: Build/%.efi # component name from the edk2 platform DSC file (such as "BiosTablesTest") is # explicit in each rule. +.NOTPARALLEL: $(foreach target,$(emulation_targets),Build/bios-tables-test.$(target).efi) + Build/bios-tables-test.%.efi: build-edk2-tools ./build.sh $(edk2_dir) BiosTablesTest $* $@ diff --git a/tests/uefi-test-tools/build.sh b/tests/uefi-test-tools/build.sh index 155cb75c4d..13891a7385 100755 --- a/tests/uefi-test-tools/build.sh +++ b/tests/uefi-test-tools/build.sh @@ -133,6 +133,7 @@ esac # Build the UEFI binary mkdir -p log build \ + -n 1 \ --arch="$edk2_arch" \ --buildtarget=DEBUG \ --platform=UefiTestToolsPkg/UefiTestToolsPkg.dsc \ --- Anyway building using "make -j1" works. [1]: Building ... qemu/roms/edk2/ArmPkg/Library/CompilerIntrinsicsLib/CompilerIntrinsicsLib.inf [ARM] make[1]: Entering directory 'qemu/tests/uefi-test-tools/Build/UefiTestTools/DEBUG_GCC5/ARM/ArmPkg/Library/CompilerIntrinsicsLib/CompilerIntrinsicsLib' "arm-linux-gnu-gcc" -mthumb -march=armv7-a -E -x assembler-with-cpp -include qemu/tests/uefi-test-tools/Build/UefiTestTools/DEBUG_GCC5/ARM/ArmPkg/Library/CompilerIntrinsicsLib/CompilerIntrinsicsLib/DEBUG/AutoGen.h -Iqemu/roms/edk2/ArmPkg/Library/CompilerIntrinsicsLib/Arm -Iqemu/roms/edk2/ArmPkg/Library/CompilerIntrinsicsLib -Iqemu/tests/uefi-test-tools/Build/UefiTestTools/DEBUG_GCC5/ARM/ArmPkg/Library/CompilerIntrinsicsLib/CompilerIntrinsicsLib/DEBUG -Iqemu/roms/edk2/MdePkg -Iqemu/roms/edk2/MdePkg/Include -Iqemu/roms/edk2/MdePkg/Include/Arm -Iqemu/roms/edk2/ArmPkg -Iqemu/roms/edk2/ArmPkg/Include qemu/roms/edk2/ArmPkg/Library/CompilerIntrinsicsLib/Arm/divsi3.S > qemu/tests/uefi-test-tools/Build/UefiTestTools/DEBUG_GCC5/ARM/ArmPkg/Library/CompilerIntrinsicsLib/CompilerIntrinsicsLib/OUTPUT/Arm/divsi3.i make[1]: *** read jobs pipe: Bad file descriptor. Stop. make[1]: *** Waiting for unfinished jobs.... build.py... : error 7000: Failed to execute command make tbuild [qemu/tests/uefi-test-tools/Build/UefiTestTools/DEBUG_GCC5/ARM/ArmPkg/Library/CompilerIntrinsicsLib/CompilerIntrinsicsLib] build.py... : error F002: Failed to build module qemu/roms/edk2/ArmPkg/Library/CompilerIntrinsicsLib/CompilerIntrinsicsLib.inf [ARM, GCC5, DEBUG] - Failed - Build end time: 13:08:01, Jan.21 2019 Build total time: 00:00:02 [2]: Active Platform = qemu/tests/uefi-test-tools/UefiTestToolsPkg/UefiTestToolsPkg.dsc Active Module = qemu/tests/uefi-test-tools/UefiTestToolsPkg/BiosTablesTest/BiosTablesTest.inf Processing meta-data . build.py... : error C0DE: Unknown fatal error when processing [qemu/roms/edk2/MdePkg/MdePkg.dec] (Please send email to edk2-devel@lists.01.org for help, attaching following call stack trace!) (Python 2.7.15 on linux2) Traceback (most recent call last): File "qemu/roms/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2494, in Main MyBuild.Launch() File "qemu/roms/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2231, in Launch self._BuildModule() File "qemu/roms/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 1837, in _BuildModule self.ModuleFile File "qemu/roms/edk2/BaseTools/Source/Python/AutoGen/AutoGen.py", line 241, in __init__ self._InitWorker(Workspace, MetaFile, Target, Toolchain, Arch, *args, **kwargs) File "qemu/roms/edk2/BaseTools/Source/Python/AutoGen/AutoGen.py", line 405, in _InitWorker PlatformPcds = Platform.Pcds File "qemu/roms/edk2/BaseTools/Source/Python/Workspace/DscBuildData.py", line 1176, in Pcds self._Pcds.update(self._GetPcd(MODEL_PCD_FIXED_AT_BUILD)) File "qemu/roms/edk2/BaseTools/Source/Python/Workspace/DscBuildData.py", line 1601, in _GetPcd PcdValue, DatumType, MaxDatumSize = self._ValidatePcd(PcdCName, TokenSpaceGuid, Setting, Type, Dummy4) File "qemu/roms/edk2/BaseTools/Source/Python/Workspace/DscBuildData.py", line 897, in _ValidatePcd self._DecPcds, self._GuidDict = GetDeclaredPcd(self, self._Bdb, self._Arch, self._Target, self._Toolchain, PkgSet) File "qemu/roms/edk2/BaseTools/Source/Python/Workspace/WorkspaceCommon.py", line 68, in GetDeclaredPcd Guids = Pkg.Guids File "qemu/roms/edk2/BaseTools/Source/Python/Workspace/DecBuildData.py", line 262, in Guids RecordList = self._RawData[MODEL_EFI_GUID, self._Arch] File "qemu/roms/edk2/BaseTools/Source/Python/Workspace/MetaFileParser.py", line 249, in __getitem__ self.Start() File "qemu/roms/edk2/BaseTools/Source/Python/Workspace/MetaFileParser.py", line 1788, in Start 0 File "qemu/roms/edk2/BaseTools/Source/Python/Workspace/MetaFileParser.py", line 195, in _Store return self._Table.Insert(*Args) File "qemu/roms/edk2/BaseTools/Source/Python/Workspace/MetaFileTable.py", line 211, in Insert Enabled File "qemu/roms/edk2/BaseTools/Source/Python/Workspace/MetaDataTable.py", line 79, in Insert self.Cur.execute(SqlCommand) IntegrityError: UNIQUE constraint failed: _1012_3.ID - Failed - Build end time: 12:44:27, Jan.21 2019 Build total time: 00:00:02 make: *** [Makefile:85: Build/bios-tables-test.x86_64.efi] Error 1 make: *** Waiting for unfinished jobs....
On 01/21/19 13:17, Philippe Mathieu-Daudé wrote: > Hi Laszlo, > > On 1/18/19 11:33 PM, Laszlo Ersek wrote: >> Introduce the following build scripts under "tests/uefi-test-tools": >> >> * "build.sh" builds a single module (a UEFI application) from >> UefiTestToolsPkg, for a single QEMU emulation target. >> >> "build.sh" relies on cross-compilers when the emulation target and the >> build host architecture don't match. The cross-compiler prefix is >> computed according to a fixed, Linux-specific pattern. No attempt is >> made to copy or reimplement the GNU Make magic from "qemu/roms/Makefile" >> for cross-compiler prefix determination. The reason is that the build >> host OSes that are officially supported by edk2, and those that are >> supported by QEMU, intersect only in Linux. (Note that the UNIXGCC >> toolchain is being removed from edk2, >> <https://bugzilla.tianocore.org/show_bug.cgi?id=1377>.) >> >> * "Makefile" currently builds the "UefiTestToolsPkg/BiosTablesTest" >> application, for arm, aarch64, i386, and x86_64, with the help of >> "build.sh". >> >> "Makefile" turns each resultant UEFI executable into a UEFI-bootable, >> qcow2-compressed ISO image. The ISO images are output as >> "tests/data/uefi-boot-images/bios-tables-test.<TARGET>.iso.qcow2". >> >> Each ISO image should be passed to QEMU as follows: >> >> -drive id=boot-cd,if=none,readonly,format=qcow2,file=$ISO \ >> -device virtio-scsi-pci,id=scsi0 \ >> -device scsi-cd,drive=boot-cd,bus=scsi0.0,bootindex=0 \ >> >> "Makefile" assumes that "mkdosfs", "mtools", and "genisoimage" are >> present. >> >> Cc: "Michael S. Tsirkin" <mst@redhat.com> >> Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org> >> Cc: Gerd Hoffmann <kraxel@redhat.com> >> Cc: Igor Mammedov <imammedo@redhat.com> >> Cc: Philippe Mathieu-Daudé <philmd@redhat.com> >> Cc: Shannon Zhao <shannon.zhaosl@gmail.com> >> Signed-off-by: Laszlo Ersek <lersek@redhat.com> >> --- >> tests/uefi-test-tools/Makefile | 92 +++++++++++++ >> tests/uefi-test-tools/.gitignore | 3 + >> tests/uefi-test-tools/build.sh | 145 ++++++++++++++++++++ >> 3 files changed, 240 insertions(+) >> >> diff --git a/tests/uefi-test-tools/Makefile b/tests/uefi-test-tools/Makefile >> new file mode 100644 >> index 000000000000..7b6dd227e433 >> --- /dev/null >> +++ b/tests/uefi-test-tools/Makefile >> @@ -0,0 +1,92 @@ >> +# Makefile for the test helper UEFI applications that run in guests. >> +# >> +# Copyright (C) 2019, Red Hat, Inc. >> +# >> +# This program and the accompanying materials are licensed and made available >> +# under the terms and conditions of the BSD License that accompanies this >> +# distribution. The full text of the license may be found at >> +# <http://opensource.org/licenses/bsd-license.php>. >> +# >> +# THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS, WITHOUT >> +# WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED. >> + >> +edk2_dir := ../../roms/edk2 >> +images_dir := ../data/uefi-boot-images >> +emulation_targets := arm aarch64 i386 x86_64 >> +uefi_binaries := bios-tables-test >> +intermediate_suffixes := .efi .fat .iso.raw >> + >> +images: $(foreach binary,$(uefi_binaries), \ >> + $(foreach target,$(emulation_targets), \ >> + $(images_dir)/$(binary).$(target).iso.qcow2)) >> + >> +# Preserve all intermediate targets if the build succeeds. >> +# - Intermediate targets help with development & debugging. >> +# - Preserving intermediate targets also keeps spurious changes out of the >> +# final build products, in case the user re-runs "make" without any changes >> +# to the UEFI source code. Normally, the intermediate files would have been >> +# removed by the last "make" invocation, hence the re-run would rebuild them >> +# from the unchanged UEFI sources. Unfortunately, the "mkdosfs" and >> +# "genisoimage" utilities embed timestamp-based information in their outputs, >> +# which causes git to report differences for the tracked qcow2 ISO images. >> +.SECONDARY: $(foreach binary,$(uefi_binaries), \ >> + $(foreach target,$(emulation_targets), \ >> + $(foreach suffix,$(intermediate_suffixes), \ >> + Build/$(binary).$(target)$(suffix)))) >> + >> +# In the pattern rules below, the stem (%, $*) stands for >> +# "$(binary).$(target)". >> + >> +# Convert the raw ISO image to a qcow2 one, enabling compression, and using a >> +# small cluster size. This allows for small binary files under git control, >> +# hence for small binary patches. >> +$(images_dir)/%.iso.qcow2: Build/%.iso.raw >> + mkdir -p -- $(images_dir) >> + $${QTEST_QEMU_IMG:-qemu-img} convert -f raw -O qcow2 -c \ >> + -o cluster_size=512 -- $< $@ >> + >> +# Embed the "UEFI system partition" into an ISO9660 file system as an ElTorito >> +# boot image. >> +Build/%.iso.raw: Build/%.fat >> + genisoimage -input-charset ASCII -efi-boot $(notdir $<) -no-emul-boot \ >> + -quiet -o $@ -- $< >> + >> +# Define chained macros in order to map QEMU system emulation targets to >> +# *short* UEFI architecture identifiers. Periods are allowed in, and ultimately >> +# stripped from, the argument. >> +map_arm_to_uefi = $(subst arm,ARM,$(1)) >> +map_aarch64_to_uefi = $(subst aarch64,AA64,$(call map_arm_to_uefi,$(1))) >> +map_i386_to_uefi = $(subst i386,IA32,$(call map_aarch64_to_uefi,$(1))) >> +map_x86_64_to_uefi = $(subst x86_64,X64,$(call map_i386_to_uefi,$(1))) >> +map_to_uefi = $(subst .,,$(call map_x86_64_to_uefi,$(1))) >> + >> +# Format a "UEFI system partition", using the UEFI binary as the default boot >> +# loader. Add 10% size for filesystem metadata, round up to the next KB, and >> +# make sure the size is large enough for a FAT filesystem. Name the filesystem >> +# after the UEFI binary. (Excess characters are automatically dropped from the >> +# filesystem label.) >> +Build/%.fat: Build/%.efi >> + rm -f -- $@ >> + uefi_bin_b=$$(stat --format=%s -- $<) && \ >> + uefi_fat_kb=$$(( (uefi_bin_b * 11 / 10 + 1023) / 1024 )) && \ >> + uefi_fat_kb=$$(( uefi_fat_kb >= 64 ? uefi_fat_kb : 64 )) && \ >> + mkdosfs -C $@ -n $(basename $(@F)) -- $$uefi_fat_kb >> + MTOOLS_SKIP_CHECK=1 mmd -i $@ ::EFI >> + MTOOLS_SKIP_CHECK=1 mmd -i $@ ::EFI/BOOT >> + MTOOLS_SKIP_CHECK=1 mcopy -i $@ -- $< \ >> + ::EFI/BOOT/BOOT$(call map_to_uefi,$(suffix $*)).EFI >> + >> +# In the pattern rules below, the stem (%, $*) stands for "$(target)" only. The >> +# association between the UEFI binary (such as "bios-tables-test") and the >> +# component name from the edk2 platform DSC file (such as "BiosTablesTest") is >> +# explicit in each rule. >> + >> +Build/bios-tables-test.%.efi: build-edk2-tools >> + ./build.sh $(edk2_dir) BiosTablesTest $* $@ >> + >> +build-edk2-tools: >> + $(MAKE) -C $(edk2_dir)/BaseTools >> + > [...] > > I got errors [1] and [2] I couldn't figure out while running 'make -j4'. > > The following patch didn't help, any idea? > > -- >8 -- > diff --git a/tests/uefi-test-tools/Makefile b/tests/uefi-test-tools/Makefile > index 7b6dd227e4..798c55c823 100644 > --- a/tests/uefi-test-tools/Makefile > +++ b/tests/uefi-test-tools/Makefile > @@ -81,6 +81,8 @@ Build/%.fat: Build/%.efi > # component name from the edk2 platform DSC file (such as > "BiosTablesTest") is > # explicit in each rule. > > +.NOTPARALLEL: $(foreach > target,$(emulation_targets),Build/bios-tables-test.$(target).efi) > + > Build/bios-tables-test.%.efi: build-edk2-tools > ./build.sh $(edk2_dir) BiosTablesTest $* $@ > > diff --git a/tests/uefi-test-tools/build.sh b/tests/uefi-test-tools/build.sh > index 155cb75c4d..13891a7385 100755 > --- a/tests/uefi-test-tools/build.sh > +++ b/tests/uefi-test-tools/build.sh > @@ -133,6 +133,7 @@ esac > # Build the UEFI binary > mkdir -p log > build \ > + -n 1 \ > --arch="$edk2_arch" \ > --buildtarget=DEBUG \ > --platform=UefiTestToolsPkg/UefiTestToolsPkg.dsc \ > --- It wasn't clear to me whether and how multi-threaded builds were supposed to be used by maintainers, whenever they'd update "tests/data/uefi-boot-images/*". I saw that "make" was invoked everywhere as $(MAKE), but that didn't clarify any intent around "-j". So I didn't test "-j" at all, and in fact I wouldn't have expected it to work: - At any point in time, there shouldn't be more than one instance of the "build" base tool working. - the "build" base tool itself contains some level of support (although not complete) for parallelizing edk2 module builds: it does not parallelize the compilation of source files between each other *within* a single edk2 module (= INF file). It parallelizes modules (INF files) between each other. - This means if you build only one module at a time, passing the "-m" switch (plus an INF file) to the "build" base tool, then there's nothing to parallelize. - "build.sh" does pass the "-m" switch to the "build" tool, because I wanted the build output to be granular. That is, it should be possible for someone to build just a given uefi-test-tools image for just a given architecture, if they specify the corresponding pathname (as target) when they invoke "make". This usage requires the "-m" flag, which in turn precludes the parallelism built into the "build" base tool itself. And, invoking multiple "build" instances in parallel is not supposed to work. Regarding why ".NOTPARALLEL" doesn't work -- the GNU Make documentation writes: [...] Any recursively invoked make command will still run recipes in parallel (unless its makefile also contains this target). [...] The "build" base tool in edk2 implements part of the job with generated makefiles and invoking "make" itself, thus, despite .NOTPARALLEL, it likely inherits the outermost -j<N> setting -- and it doesn't expect such. So the best I can offer here is to check $MAKEFLAGS in "build.sh", and exit with an early, explicit error if $MAKEFLAGS contains "-j", "-l", or their long variants (--jobs, --load-average). I realize this would not be acceptable for building QEMU itself, but "tests/uefi-test-tools/Makefile" should be run by some subsystem maintainers only, interactively (or in their on custom scripts). Thanks Laszlo
On Mon, 21 Jan 2019 at 19:09, Laszlo Ersek <lersek@redhat.com> wrote: > It wasn't clear to me whether and how multi-threaded builds were > supposed to be used by maintainers, whenever they'd update > "tests/data/uefi-boot-images/*". > > I saw that "make" was invoked everywhere as $(MAKE), but that didn't > clarify any intent around "-j". So I didn't test "-j" at all, and in > fact I wouldn't have expected it to work: The usual assumption with make is that "-jN" should work and at least (if the thing being built can't actually be parallelized usefully) be no worse than if you'd not specified a -j option. We have occasionally had problems with -jN in 'make check' (usually because several test cases were trying to use the same temp filename or similar) but we've treated them as bugs and squashed them. > The "build" base tool in edk2 implements part of the job with generated > makefiles and invoking "make" itself, thus, despite .NOTPARALLEL, it > likely inherits the outermost -j<N> setting -- and it doesn't expect such. > > So the best I can offer here is to check $MAKEFLAGS in "build.sh", and > exit with an early, explicit error if $MAKEFLAGS contains "-j", "-l", or > their long variants (--jobs, --load-average). Could you sanitize MAKEFLAGS in build.sh instead to remove the parallelization options? thanks -- PMM
On 01/21/19 20:30, Peter Maydell wrote: > On Mon, 21 Jan 2019 at 19:09, Laszlo Ersek <lersek@redhat.com> wrote: >> It wasn't clear to me whether and how multi-threaded builds were >> supposed to be used by maintainers, whenever they'd update >> "tests/data/uefi-boot-images/*". >> >> I saw that "make" was invoked everywhere as $(MAKE), but that didn't >> clarify any intent around "-j". So I didn't test "-j" at all, and in >> fact I wouldn't have expected it to work: > > The usual assumption with make is that "-jN" should work > and at least (if the thing being built can't actually > be parallelized usefully) be no worse than if you'd not > specified a -j option. We have occasionally had problems > with -jN in 'make check' (usually because several test cases > were trying to use the same temp filename or similar) but > we've treated them as bugs and squashed them. > >> The "build" base tool in edk2 implements part of the job with generated >> makefiles and invoking "make" itself, thus, despite .NOTPARALLEL, it >> likely inherits the outermost -j<N> setting -- and it doesn't expect such. >> >> So the best I can offer here is to check $MAKEFLAGS in "build.sh", and >> exit with an early, explicit error if $MAKEFLAGS contains "-j", "-l", or >> their long variants (--jobs, --load-average). > > Could you sanitize MAKEFLAGS in build.sh instead to remove the > parallelization options? I've looked into MAKEFLAGS in a bit more depth now; both the documentation and some debug prints. Manipulating MAKEFLAGS looks somewhat brittle. So, I was about to suggest that I use the .NOTPARALLEL special target recommended by Phil, for the .efi binaries (which would ensure that no two instances of the "build" base tool run at the same time), *plus* that I submit a patch to edk2 so that the makefiles generated by the "build" tool also contain .NOTPARALLEL. (Because, to quote the make docs again, "If .NOTPARALLEL is mentioned as a target, then this invocation of make will be run serially, even if the ‘-j’ option is given. Any recursively invoked make command will still run recipes in parallel (unless its makefile also contains this target).") However: when I wanted to see the actual error from using .NOTPARALLEL *only* in "tests/uefi-test-tools/Makefile", and not in the build-generated makefiles, I failed to get any error. All the output images were built just fine. Phil: when you wrote that "The following patch didn't help" -- referring to .NOTPARALLEL, added only to "tests/uefi-test-tools/Makefile"-- , did you clean your tree first (with "make clean" or "git clean -ffdx")? Because now I'm thinking that the *individual* makefiles generated by edk2's "build" base tool might actually compatible with "-j", and your testing of .NOTPARALLEL failed only because your build tree (for example the Conf/ subdir -- list it with "-A") was in a messy state from your previous -j4 attempt (where you didn't use .NOTPARALLEL at all). Thanks Laszlo
On 01/22/19 13:02, Laszlo Ersek wrote: > On 01/21/19 20:30, Peter Maydell wrote: >> On Mon, 21 Jan 2019 at 19:09, Laszlo Ersek <lersek@redhat.com> wrote: >>> It wasn't clear to me whether and how multi-threaded builds were >>> supposed to be used by maintainers, whenever they'd update >>> "tests/data/uefi-boot-images/*". >>> >>> I saw that "make" was invoked everywhere as $(MAKE), but that didn't >>> clarify any intent around "-j". So I didn't test "-j" at all, and in >>> fact I wouldn't have expected it to work: >> >> The usual assumption with make is that "-jN" should work >> and at least (if the thing being built can't actually >> be parallelized usefully) be no worse than if you'd not >> specified a -j option. We have occasionally had problems >> with -jN in 'make check' (usually because several test cases >> were trying to use the same temp filename or similar) but >> we've treated them as bugs and squashed them. >> >>> The "build" base tool in edk2 implements part of the job with generated >>> makefiles and invoking "make" itself, thus, despite .NOTPARALLEL, it >>> likely inherits the outermost -j<N> setting -- and it doesn't expect such. >>> >>> So the best I can offer here is to check $MAKEFLAGS in "build.sh", and >>> exit with an early, explicit error if $MAKEFLAGS contains "-j", "-l", or >>> their long variants (--jobs, --load-average). >> >> Could you sanitize MAKEFLAGS in build.sh instead to remove the >> parallelization options? > > I've looked into MAKEFLAGS in a bit more depth now; both the > documentation and some debug prints. Manipulating MAKEFLAGS looks > somewhat brittle. > > So, I was about to suggest that I use the .NOTPARALLEL special target > recommended by Phil, for the .efi binaries (which would ensure that no > two instances of the "build" base tool run at the same time), *plus* > that I submit a patch to edk2 so that the makefiles generated by the > "build" tool also contain .NOTPARALLEL. > > (Because, to quote the make docs again, "If .NOTPARALLEL is mentioned as > a target, then this invocation of make will be run serially, even if the > ‘-j’ option is given. Any recursively invoked make command will still > run recipes in parallel (unless its makefile also contains this target).") > > However: when I wanted to see the actual error from using .NOTPARALLEL > *only* in "tests/uefi-test-tools/Makefile", and not in the > build-generated makefiles, I failed to get any error. All the output > images were built just fine. > > Phil: when you wrote that "The following patch didn't help" -- referring > to .NOTPARALLEL, added only to "tests/uefi-test-tools/Makefile"-- , did > you clean your tree first (with "make clean" or "git clean -ffdx")? > > Because now I'm thinking that the *individual* makefiles generated by > edk2's "build" base tool might actually compatible with "-j", and your > testing of .NOTPARALLEL failed only because your build tree (for example > the Conf/ subdir -- list it with "-A") was in a messy state from your > previous -j4 attempt (where you didn't use .NOTPARALLEL at all). FWIW, I've asked - both on edk2-devel, about adding .NOTPARALLEL to the generated inner makefiles: "parallelism in the module-level, generated GNUmakefile's" https://lists.01.org/pipermail/edk2-devel/2019-January/035463.html - and on help-make, about safely filtering the job-related options from MAKEFLAGS, between the outer and inner "make" processes: "filtering job options from MAKEFLAGS, manually" http://lists.gnu.org/archive/html/help-make/2019-01/msg00003.html Thanks Laszlo
On 01/23/19 17:13, Laszlo Ersek wrote: > FWIW, I've asked > > - both on edk2-devel, about adding .NOTPARALLEL to the generated inner > makefiles: > > "parallelism in the module-level, generated GNUmakefile's" > https://lists.01.org/pipermail/edk2-devel/2019-January/035463.html > > - and on help-make, about safely filtering the job-related options from > MAKEFLAGS, between the outer and inner "make" processes: > > "filtering job options from MAKEFLAGS, manually" > http://lists.gnu.org/archive/html/help-make/2019-01/msg00003.html Results: - The "build" utility in BaseTools generates such GNUMakefiles that dependencies between targets are fully described. This is an explicit goal and if we notice any missing deps, that's a bug to be reported and fixed. As a consequence, we can expect the recipes in those GNUMakefiles to be fully parallelizable. Therefore we need not add .NOTPARALLEL to them (in the edk2 project), or filter MAKEFLAGS, or append "-j1" to MAKEFLAGS. - The .NOTPARALLEL target ignores any pre-requisites it gets; .NOTPARALLEL applies to the entire current invocation of make. Therefore we just need to stick a bare .NOTPARALLEL into "tests/uefi-test-tools/Makefile", to keep "build" itself isolated. I'm going to post v2 with this one addition (i.e., .NOTPARALLEL). Thanks, Laszlo
diff --git a/tests/uefi-test-tools/Makefile b/tests/uefi-test-tools/Makefile new file mode 100644 index 000000000000..7b6dd227e433 --- /dev/null +++ b/tests/uefi-test-tools/Makefile @@ -0,0 +1,92 @@ +# Makefile for the test helper UEFI applications that run in guests. +# +# Copyright (C) 2019, Red Hat, Inc. +# +# This program and the accompanying materials are licensed and made available +# under the terms and conditions of the BSD License that accompanies this +# distribution. The full text of the license may be found at +# <http://opensource.org/licenses/bsd-license.php>. +# +# THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS, WITHOUT +# WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED. + +edk2_dir := ../../roms/edk2 +images_dir := ../data/uefi-boot-images +emulation_targets := arm aarch64 i386 x86_64 +uefi_binaries := bios-tables-test +intermediate_suffixes := .efi .fat .iso.raw + +images: $(foreach binary,$(uefi_binaries), \ + $(foreach target,$(emulation_targets), \ + $(images_dir)/$(binary).$(target).iso.qcow2)) + +# Preserve all intermediate targets if the build succeeds. +# - Intermediate targets help with development & debugging. +# - Preserving intermediate targets also keeps spurious changes out of the +# final build products, in case the user re-runs "make" without any changes +# to the UEFI source code. Normally, the intermediate files would have been +# removed by the last "make" invocation, hence the re-run would rebuild them +# from the unchanged UEFI sources. Unfortunately, the "mkdosfs" and +# "genisoimage" utilities embed timestamp-based information in their outputs, +# which causes git to report differences for the tracked qcow2 ISO images. +.SECONDARY: $(foreach binary,$(uefi_binaries), \ + $(foreach target,$(emulation_targets), \ + $(foreach suffix,$(intermediate_suffixes), \ + Build/$(binary).$(target)$(suffix)))) + +# In the pattern rules below, the stem (%, $*) stands for +# "$(binary).$(target)". + +# Convert the raw ISO image to a qcow2 one, enabling compression, and using a +# small cluster size. This allows for small binary files under git control, +# hence for small binary patches. +$(images_dir)/%.iso.qcow2: Build/%.iso.raw + mkdir -p -- $(images_dir) + $${QTEST_QEMU_IMG:-qemu-img} convert -f raw -O qcow2 -c \ + -o cluster_size=512 -- $< $@ + +# Embed the "UEFI system partition" into an ISO9660 file system as an ElTorito +# boot image. +Build/%.iso.raw: Build/%.fat + genisoimage -input-charset ASCII -efi-boot $(notdir $<) -no-emul-boot \ + -quiet -o $@ -- $< + +# Define chained macros in order to map QEMU system emulation targets to +# *short* UEFI architecture identifiers. Periods are allowed in, and ultimately +# stripped from, the argument. +map_arm_to_uefi = $(subst arm,ARM,$(1)) +map_aarch64_to_uefi = $(subst aarch64,AA64,$(call map_arm_to_uefi,$(1))) +map_i386_to_uefi = $(subst i386,IA32,$(call map_aarch64_to_uefi,$(1))) +map_x86_64_to_uefi = $(subst x86_64,X64,$(call map_i386_to_uefi,$(1))) +map_to_uefi = $(subst .,,$(call map_x86_64_to_uefi,$(1))) + +# Format a "UEFI system partition", using the UEFI binary as the default boot +# loader. Add 10% size for filesystem metadata, round up to the next KB, and +# make sure the size is large enough for a FAT filesystem. Name the filesystem +# after the UEFI binary. (Excess characters are automatically dropped from the +# filesystem label.) +Build/%.fat: Build/%.efi + rm -f -- $@ + uefi_bin_b=$$(stat --format=%s -- $<) && \ + uefi_fat_kb=$$(( (uefi_bin_b * 11 / 10 + 1023) / 1024 )) && \ + uefi_fat_kb=$$(( uefi_fat_kb >= 64 ? uefi_fat_kb : 64 )) && \ + mkdosfs -C $@ -n $(basename $(@F)) -- $$uefi_fat_kb + MTOOLS_SKIP_CHECK=1 mmd -i $@ ::EFI + MTOOLS_SKIP_CHECK=1 mmd -i $@ ::EFI/BOOT + MTOOLS_SKIP_CHECK=1 mcopy -i $@ -- $< \ + ::EFI/BOOT/BOOT$(call map_to_uefi,$(suffix $*)).EFI + +# In the pattern rules below, the stem (%, $*) stands for "$(target)" only. The +# association between the UEFI binary (such as "bios-tables-test") and the +# component name from the edk2 platform DSC file (such as "BiosTablesTest") is +# explicit in each rule. + +Build/bios-tables-test.%.efi: build-edk2-tools + ./build.sh $(edk2_dir) BiosTablesTest $* $@ + +build-edk2-tools: + $(MAKE) -C $(edk2_dir)/BaseTools + +clean: + rm -rf Build Conf log + $(MAKE) -C $(edk2_dir)/BaseTools clean diff --git a/tests/uefi-test-tools/.gitignore b/tests/uefi-test-tools/.gitignore new file mode 100644 index 000000000000..9f246701dea1 --- /dev/null +++ b/tests/uefi-test-tools/.gitignore @@ -0,0 +1,3 @@ +Build +Conf +log diff --git a/tests/uefi-test-tools/build.sh b/tests/uefi-test-tools/build.sh new file mode 100755 index 000000000000..155cb75c4ddb --- /dev/null +++ b/tests/uefi-test-tools/build.sh @@ -0,0 +1,145 @@ +#!/bin/bash + +# Build script that determines the edk2 toolchain to use, invokes the edk2 +# "build" utility, and copies the built UEFI binary to the requested location. +# +# Copyright (C) 2019, Red Hat, Inc. +# +# This program and the accompanying materials are licensed and made available +# under the terms and conditions of the BSD License that accompanies this +# distribution. The full text of the license may be found at +# <http://opensource.org/licenses/bsd-license.php>. +# +# THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS, WITHOUT +# WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED. + +set -e -u -C + +# Save the command line arguments. We need to reset $# to 0 before sourcing +# "edksetup.sh", as it will inherit $@. +program_name=$(basename -- "$0") +edk2_dir=$1 +dsc_component=$2 +emulation_target=$3 +uefi_binary=$4 +shift 4 + +# Set up the environment for edk2 building. +export PACKAGES_PATH=$(realpath -- "$edk2_dir") +export WORKSPACE=$PWD +mkdir -p Conf + +# Source "edksetup.sh" carefully. +set +e +u +C +source "$PACKAGES_PATH/edksetup.sh" +ret=$? +set -e -u -C +if [ $ret -ne 0 ]; then + exit $ret +fi + +# Map the QEMU system emulation target to the following types of architecture +# identifiers: +# - edk2, +# - gcc cross-compilation. +# Cover only those targets that are supported by the UEFI spec and edk2. +case "$emulation_target" in + (arm) + edk2_arch=ARM + gcc_arch=arm + ;; + (aarch64) + edk2_arch=AARCH64 + gcc_arch=aarch64 + ;; + (i386) + edk2_arch=IA32 + gcc_arch=i686 + ;; + (x86_64) + edk2_arch=X64 + gcc_arch=x86_64 + ;; + (*) + printf '%s: unknown/unsupported QEMU system emulation target "%s"\n' \ + "$program_name" "$emulation_target" >&2 + exit 1 + ;; +esac + +# Check if cross-compilation is needed. +host_arch=$(uname -m) +if [ "$gcc_arch" == "$host_arch" ] || + ( [ "$gcc_arch" == i686 ] && [ "$host_arch" == x86_64 ] ); then + cross_prefix= +else + cross_prefix=${gcc_arch}-linux-gnu- +fi + +# Expose cross_prefix (which is possibly empty) to the edk2 tools. While at it, +# determine the suitable edk2 toolchain as well. +# - For ARM and AARCH64, edk2 only offers the GCC5 toolchain tag, which covers +# the gcc-5+ releases. +# - For IA32 and X64, edk2 offers the GCC44 through GCC49 toolchain tags, in +# addition to GCC5. Unfortunately, the mapping between the toolchain tags and +# the actual gcc releases isn't entirely trivial. Run "git-blame" on +# "OvmfPkg/build.sh" in edk2 for more information. +# And, because the above is too simple, we have to assign cross_prefix to an +# edk2 build variable that is specific to both the toolchain tag and the target +# architecture. +case "$edk2_arch" in + (ARM) + edk2_toolchain=GCC5 + export GCC5_ARM_PREFIX=$cross_prefix + ;; + (AARCH64) + edk2_toolchain=GCC5 + export GCC5_AARCH64_PREFIX=$cross_prefix + ;; + (IA32|X64) + gcc_version=$("${cross_prefix}gcc" -v 2>&1 | tail -1 | awk '{print $3}') + case "$gcc_version" in + ([1-3].*|4.[0-3].*) + printf '%s: unsupported gcc version "%s"\n' \ + "$program_name" "$gcc_version" >&2 + exit 1 + ;; + (4.4.*) + edk2_toolchain=GCC44 + ;; + (4.5.*) + edk2_toolchain=GCC45 + ;; + (4.6.*) + edk2_toolchain=GCC46 + ;; + (4.7.*) + edk2_toolchain=GCC47 + ;; + (4.8.*) + edk2_toolchain=GCC48 + ;; + (4.9.*|6.[0-2].*) + edk2_toolchain=GCC49 + ;; + (*) + edk2_toolchain=GCC5 + ;; + esac + eval "export ${edk2_toolchain}_BIN=\$cross_prefix" + ;; +esac + +# Build the UEFI binary +mkdir -p log +build \ + --arch="$edk2_arch" \ + --buildtarget=DEBUG \ + --platform=UefiTestToolsPkg/UefiTestToolsPkg.dsc \ + --tagname="$edk2_toolchain" \ + --module="UefiTestToolsPkg/$dsc_component/$dsc_component.inf" \ + --log="log/$dsc_component.$edk2_arch.log" \ + --report-file="log/$dsc_component.$edk2_arch.report" +cp -a -- \ + "Build/UefiTestTools/DEBUG_${edk2_toolchain}/$edk2_arch/$dsc_component.efi" \ + "$uefi_binary"
Introduce the following build scripts under "tests/uefi-test-tools": * "build.sh" builds a single module (a UEFI application) from UefiTestToolsPkg, for a single QEMU emulation target. "build.sh" relies on cross-compilers when the emulation target and the build host architecture don't match. The cross-compiler prefix is computed according to a fixed, Linux-specific pattern. No attempt is made to copy or reimplement the GNU Make magic from "qemu/roms/Makefile" for cross-compiler prefix determination. The reason is that the build host OSes that are officially supported by edk2, and those that are supported by QEMU, intersect only in Linux. (Note that the UNIXGCC toolchain is being removed from edk2, <https://bugzilla.tianocore.org/show_bug.cgi?id=1377>.) * "Makefile" currently builds the "UefiTestToolsPkg/BiosTablesTest" application, for arm, aarch64, i386, and x86_64, with the help of "build.sh". "Makefile" turns each resultant UEFI executable into a UEFI-bootable, qcow2-compressed ISO image. The ISO images are output as "tests/data/uefi-boot-images/bios-tables-test.<TARGET>.iso.qcow2". Each ISO image should be passed to QEMU as follows: -drive id=boot-cd,if=none,readonly,format=qcow2,file=$ISO \ -device virtio-scsi-pci,id=scsi0 \ -device scsi-cd,drive=boot-cd,bus=scsi0.0,bootindex=0 \ "Makefile" assumes that "mkdosfs", "mtools", and "genisoimage" are present. Cc: "Michael S. Tsirkin" <mst@redhat.com> Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org> Cc: Gerd Hoffmann <kraxel@redhat.com> Cc: Igor Mammedov <imammedo@redhat.com> Cc: Philippe Mathieu-Daudé <philmd@redhat.com> Cc: Shannon Zhao <shannon.zhaosl@gmail.com> Signed-off-by: Laszlo Ersek <lersek@redhat.com> --- tests/uefi-test-tools/Makefile | 92 +++++++++++++ tests/uefi-test-tools/.gitignore | 3 + tests/uefi-test-tools/build.sh | 145 ++++++++++++++++++++ 3 files changed, 240 insertions(+)