[00/10] bundle edk2 platform firmware with QEMU

Message ID	20190309004826.9027-1-lersek@redhat.com
Headers	show Return-Path: <qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org> From: Laszlo Ersek <lersek@redhat.com> To: berrange@redhat.com, imammedo@redhat.com, kraxel@redhat.com, mst@redhat.com, philmd@redhat.com, qemu-devel@nongnu.org Date: Sat, 9 Mar 2019 01:48:16 +0100 Message-Id: <20190309004826.9027-1-lersek@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Subject: [Qemu-devel] [PATCH 00/10] bundle edk2 platform firmware with QEMU Precedence: list Errors-To: qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org Sender: "Qemu-devel" <qemu-devel-bounces+incoming=patchwork.ozlabs.org@nongnu.org>
Series	bundle edk2 platform firmware with QEMU \| expand [00/10] bundle edk2 platform firmware with QEMU [01/10] roms: lift "edk2-funcs.sh" from "tests/uefi-test-tools/build.sh" [02/10] roms/edk2-funcs.sh: require gcc-4.8+ for building i386 and x86_64 [03/10] tests/uefi-test-tools/build.sh: work around TianoCore#1607 [04/10] roms/edk2: advance to tag edk2-stable201903 [05/10] roms/edk2-funcs.sh: add the qemu_edk2_get_thread_count() function [06/10] roms/Makefile: replace the $(EFIROM) target with "edk2-basetools" [07/10] roms: build edk2 firmware binaries and variable store templates [08/10] pc-bios: add edk2 firmware binaries and variable store templates [09/10] pc-bios: document the edk2 firmware images; add firmware descriptors [10/10] Makefile: install the edk2 firmware images and their descriptors

Laszlo Ersek March 9, 2019, 12:48 a.m. UTC

Repo:   https://github.com/lersek/qemu.git
Branch: edk2_build

This series advances the roms/edk2 submodule to the "edk2-stable201903"
release, and builds and captures platform firmware binaries from that
release. At this point they are meant to be used by both end-users and
by Igor's ACPI unit tests in qtest ("make check").

Previous discussion:

  [Qemu-devel] bundling edk2 platform firmware images with QEMU
  http://mid.mail-archive.com/80f0bae3-e79a-bb68-04c4-1c9c684d95b8@redhat.com
  https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg02601.html

Note that the series was formatted with "--no-binary" (affecting patch
#8), therefore it cannot be applied with "git-am". See the remote
repo/branch reference near the top instead.

Thanks,
Laszlo

Laszlo Ersek (10):
  roms: lift "edk2-funcs.sh" from "tests/uefi-test-tools/build.sh"
  roms/edk2-funcs.sh: require gcc-4.8+ for building i386 and x86_64
  tests/uefi-test-tools/build.sh: work around TianoCore#1607
  roms/edk2: advance to tag edk2-stable201903
  roms/edk2-funcs.sh: add the qemu_edk2_get_thread_count() function
  roms/Makefile: replace the $(EFIROM) target with "edk2-basetools"
  roms: build edk2 firmware binaries and variable store templates
  pc-bios: add edk2 firmware binaries and variable store templates
  pc-bios: document the edk2 firmware images; add firmware descriptors
  Makefile: install the edk2 firmware images and their descriptors

 Makefile                                       |  17 +-
 pc-bios/README                                 |  11 +
 pc-bios/descriptors/50-edk2-i386-secure.json   |  34 +++
 pc-bios/descriptors/50-edk2-x86_64-secure.json |  35 +++
 pc-bios/descriptors/60-edk2-aarch64.json       |  31 +++
 pc-bios/descriptors/60-edk2-arm.json           |  31 +++
 pc-bios/descriptors/60-edk2-i386.json          |  33 +++
 pc-bios/descriptors/60-edk2-x86_64.json        |  34 +++
 pc-bios/edk2-aarch64-code.fd                   | Bin 0 -> 67108864 bytes
 pc-bios/edk2-arm-code.fd                       | Bin 0 -> 67108864 bytes
 pc-bios/edk2-arm-vars.fd                       | Bin 0 -> 67108864 bytes
 pc-bios/edk2-i386-code.fd                      | Bin 0 -> 3653632 bytes
 pc-bios/edk2-i386-secure-code.fd               | Bin 0 -> 3653632 bytes
 pc-bios/edk2-i386-vars.fd                      | Bin 0 -> 540672 bytes
 pc-bios/edk2-licenses.txt                      | 209 ++++++++++++++++
 pc-bios/edk2-x86_64-code.fd                    | Bin 0 -> 3653632 bytes
 pc-bios/edk2-x86_64-secure-code.fd             | Bin 0 -> 3653632 bytes
 roms/Makefile                                  |   9 +-
 roms/Makefile.edk2                             | 138 +++++++++++
 roms/edk2                                      |   2 +-
 roms/edk2-build.sh                             |  55 +++++
 roms/edk2-funcs.sh                             | 253 ++++++++++++++++++++
 tests/uefi-test-tools/build.sh                 | 100 +-------
 23 files changed, 897 insertions(+), 95 deletions(-)
 create mode 100644 pc-bios/descriptors/50-edk2-i386-secure.json
 create mode 100644 pc-bios/descriptors/50-edk2-x86_64-secure.json
 create mode 100644 pc-bios/descriptors/60-edk2-aarch64.json
 create mode 100644 pc-bios/descriptors/60-edk2-arm.json
 create mode 100644 pc-bios/descriptors/60-edk2-i386.json
 create mode 100644 pc-bios/descriptors/60-edk2-x86_64.json
 create mode 100644 pc-bios/edk2-aarch64-code.fd
 create mode 100644 pc-bios/edk2-arm-code.fd
 create mode 100644 pc-bios/edk2-arm-vars.fd
 create mode 100644 pc-bios/edk2-i386-code.fd
 create mode 100644 pc-bios/edk2-i386-secure-code.fd
 create mode 100644 pc-bios/edk2-i386-vars.fd
 create mode 100644 pc-bios/edk2-licenses.txt
 create mode 100644 pc-bios/edk2-x86_64-code.fd
 create mode 100644 pc-bios/edk2-x86_64-secure-code.fd
 create mode 100644 roms/Makefile.edk2
 create mode 100755 roms/edk2-build.sh
 create mode 100644 roms/edk2-funcs.sh

Philippe Mathieu-Daudé March 9, 2019, 1:20 a.m. UTC | #1

On 3/9/19 1:48 AM, Laszlo Ersek wrote:
> Repo:   https://github.com/lersek/qemu.git
> Branch: edk2_build
> 
> This series advances the roms/edk2 submodule to the "edk2-stable201903"
> release, and builds and captures platform firmware binaries from that
> release. At this point they are meant to be used by both end-users and
> by Igor's ACPI unit tests in qtest ("make check").
> 
> Previous discussion:
> 
>   [Qemu-devel] bundling edk2 platform firmware images with QEMU
>   http://mid.mail-archive.com/80f0bae3-e79a-bb68-04c4-1c9c684d95b8@redhat.com
>   https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg02601.html
> 
> Note that the series was formatted with "--no-binary" (affecting patch
> #8), therefore it cannot be applied with "git-am". See the remote
> repo/branch reference near the top instead.
> 
> Thanks,
> Laszlo
> 
> Laszlo Ersek (10):
>   roms: lift "edk2-funcs.sh" from "tests/uefi-test-tools/build.sh"
>   roms/edk2-funcs.sh: require gcc-4.8+ for building i386 and x86_64
>   tests/uefi-test-tools/build.sh: work around TianoCore#1607
>   roms/edk2: advance to tag edk2-stable201903
>   roms/edk2-funcs.sh: add the qemu_edk2_get_thread_count() function
>   roms/Makefile: replace the $(EFIROM) target with "edk2-basetools"
>   roms: build edk2 firmware binaries and variable store templates
>   pc-bios: add edk2 firmware binaries and variable store templates
>   pc-bios: document the edk2 firmware images; add firmware descriptors
>   Makefile: install the edk2 firmware images and their descriptors
> 
>  Makefile                                       |  17 +-
>  pc-bios/README                                 |  11 +
>  pc-bios/descriptors/50-edk2-i386-secure.json   |  34 +++
>  pc-bios/descriptors/50-edk2-x86_64-secure.json |  35 +++
>  pc-bios/descriptors/60-edk2-aarch64.json       |  31 +++
>  pc-bios/descriptors/60-edk2-arm.json           |  31 +++
>  pc-bios/descriptors/60-edk2-i386.json          |  33 +++
>  pc-bios/descriptors/60-edk2-x86_64.json        |  34 +++
>  pc-bios/edk2-aarch64-code.fd                   | Bin 0 -> 67108864 bytes
>  pc-bios/edk2-arm-code.fd                       | Bin 0 -> 67108864 bytes
>  pc-bios/edk2-arm-vars.fd                       | Bin 0 -> 67108864 bytes

GitHub moans here:

remote: warning: GH001: Large files detected. You may want to try Git
Large File Storage - https://git-lfs.github.com.
remote: warning: See http://git.io/iEPt8g for more information.
remote: warning: File pc-bios/edk2-arm-vars.fd is 64.00 MB; this is
larger than GitHub's recommended maximum file size of 50.00 MB
remote: warning: File pc-bios/edk2-arm-code.fd is 64.00 MB; this is
larger than GitHub's recommended maximum file size of 50.00 MB
remote: warning: File pc-bios/edk2-aarch64-code.fd is 64.00 MB; this is
larger than GitHub's recommended maximum file size of 50.00 MB

>  pc-bios/edk2-i386-code.fd                      | Bin 0 -> 3653632 bytes
>  pc-bios/edk2-i386-secure-code.fd               | Bin 0 -> 3653632 bytes
>  pc-bios/edk2-i386-vars.fd                      | Bin 0 -> 540672 bytes
>  pc-bios/edk2-licenses.txt                      | 209 ++++++++++++++++
>  pc-bios/edk2-x86_64-code.fd                    | Bin 0 -> 3653632 bytes
>  pc-bios/edk2-x86_64-secure-code.fd             | Bin 0 -> 3653632 bytes
>  roms/Makefile                                  |   9 +-
>  roms/Makefile.edk2                             | 138 +++++++++++
>  roms/edk2                                      |   2 +-
>  roms/edk2-build.sh                             |  55 +++++
>  roms/edk2-funcs.sh                             | 253 ++++++++++++++++++++
>  tests/uefi-test-tools/build.sh                 | 100 +-------
>  23 files changed, 897 insertions(+), 95 deletions(-)
>  create mode 100644 pc-bios/descriptors/50-edk2-i386-secure.json
>  create mode 100644 pc-bios/descriptors/50-edk2-x86_64-secure.json
>  create mode 100644 pc-bios/descriptors/60-edk2-aarch64.json
>  create mode 100644 pc-bios/descriptors/60-edk2-arm.json
>  create mode 100644 pc-bios/descriptors/60-edk2-i386.json
>  create mode 100644 pc-bios/descriptors/60-edk2-x86_64.json
>  create mode 100644 pc-bios/edk2-aarch64-code.fd
>  create mode 100644 pc-bios/edk2-arm-code.fd
>  create mode 100644 pc-bios/edk2-arm-vars.fd
>  create mode 100644 pc-bios/edk2-i386-code.fd
>  create mode 100644 pc-bios/edk2-i386-secure-code.fd
>  create mode 100644 pc-bios/edk2-i386-vars.fd
>  create mode 100644 pc-bios/edk2-licenses.txt
>  create mode 100644 pc-bios/edk2-x86_64-code.fd
>  create mode 100644 pc-bios/edk2-x86_64-secure-code.fd
>  create mode 100644 roms/Makefile.edk2
>  create mode 100755 roms/edk2-build.sh
>  create mode 100644 roms/edk2-funcs.sh
>

Michal Prívozník March 9, 2019, 1:10 p.m. UTC | #2

On 3/9/19 1:48 AM, Laszlo Ersek wrote:
> Repo:   https://github.com/lersek/qemu.git
> Branch: edk2_build
> 
> This series advances the roms/edk2 submodule to the "edk2-stable201903"
> release, and builds and captures platform firmware binaries from that
> release. At this point they are meant to be used by both end-users and
> by Igor's ACPI unit tests in qtest ("make check").
> 
> Previous discussion:
> 
>   [Qemu-devel] bundling edk2 platform firmware images with QEMU
>   http://mid.mail-archive.com/80f0bae3-e79a-bb68-04c4-1c9c684d95b8@redhat.com
>   https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg02601.html
> 
> Note that the series was formatted with "--no-binary" (affecting patch
> #8), therefore it cannot be applied with "git-am". See the remote
> repo/branch reference near the top instead.
> 
> Thanks,
> Laszlo
> 
> Laszlo Ersek (10):
>   roms: lift "edk2-funcs.sh" from "tests/uefi-test-tools/build.sh"
>   roms/edk2-funcs.sh: require gcc-4.8+ for building i386 and x86_64
>   tests/uefi-test-tools/build.sh: work around TianoCore#1607
>   roms/edk2: advance to tag edk2-stable201903
>   roms/edk2-funcs.sh: add the qemu_edk2_get_thread_count() function
>   roms/Makefile: replace the $(EFIROM) target with "edk2-basetools"
>   roms: build edk2 firmware binaries and variable store templates
>   pc-bios: add edk2 firmware binaries and variable store templates
>   pc-bios: document the edk2 firmware images; add firmware descriptors
>   Makefile: install the edk2 firmware images and their descriptors
> 
>  Makefile                                       |  17 +-
>  pc-bios/README                                 |  11 +
>  pc-bios/descriptors/50-edk2-i386-secure.json   |  34 +++
>  pc-bios/descriptors/50-edk2-x86_64-secure.json |  35 +++
>  pc-bios/descriptors/60-edk2-aarch64.json       |  31 +++
>  pc-bios/descriptors/60-edk2-arm.json           |  31 +++
>  pc-bios/descriptors/60-edk2-i386.json          |  33 +++
>  pc-bios/descriptors/60-edk2-x86_64.json        |  34 +++
>  pc-bios/edk2-aarch64-code.fd                   | Bin 0 -> 67108864 bytes
>  pc-bios/edk2-arm-code.fd                       | Bin 0 -> 67108864 bytes
>  pc-bios/edk2-arm-vars.fd                       | Bin 0 -> 67108864 bytes
>  pc-bios/edk2-i386-code.fd                      | Bin 0 -> 3653632 bytes
>  pc-bios/edk2-i386-secure-code.fd               | Bin 0 -> 3653632 bytes
>  pc-bios/edk2-i386-vars.fd                      | Bin 0 -> 540672 bytes
>  pc-bios/edk2-licenses.txt                      | 209 ++++++++++++++++
>  pc-bios/edk2-x86_64-code.fd                    | Bin 0 -> 3653632 bytes
>  pc-bios/edk2-x86_64-secure-code.fd             | Bin 0 -> 3653632 bytes
>  roms/Makefile                                  |   9 +-
>  roms/Makefile.edk2                             | 138 +++++++++++
>  roms/edk2                                      |   2 +-
>  roms/edk2-build.sh                             |  55 +++++
>  roms/edk2-funcs.sh                             | 253 ++++++++++++++++++++
>  tests/uefi-test-tools/build.sh                 | 100 +-------
>  23 files changed, 897 insertions(+), 95 deletions(-)
>  create mode 100644 pc-bios/descriptors/50-edk2-i386-secure.json
>  create mode 100644 pc-bios/descriptors/50-edk2-x86_64-secure.json
>  create mode 100644 pc-bios/descriptors/60-edk2-aarch64.json
>  create mode 100644 pc-bios/descriptors/60-edk2-arm.json
>  create mode 100644 pc-bios/descriptors/60-edk2-i386.json
>  create mode 100644 pc-bios/descriptors/60-edk2-x86_64.json
>  create mode 100644 pc-bios/edk2-aarch64-code.fd
>  create mode 100644 pc-bios/edk2-arm-code.fd
>  create mode 100644 pc-bios/edk2-arm-vars.fd
>  create mode 100644 pc-bios/edk2-i386-code.fd
>  create mode 100644 pc-bios/edk2-i386-secure-code.fd
>  create mode 100644 pc-bios/edk2-i386-vars.fd
>  create mode 100644 pc-bios/edk2-licenses.txt
>  create mode 100644 pc-bios/edk2-x86_64-code.fd
>  create mode 100644 pc-bios/edk2-x86_64-secure-code.fd
>  create mode 100644 roms/Makefile.edk2
>  create mode 100755 roms/edk2-build.sh
>  create mode 100644 roms/edk2-funcs.sh
> 

Unsure whether my ACK is worth anything on this list, but you have it.

Reviewed-by: Michal Privoznik <mprivozn@redhat.com>

Now that libvirt patches for consuming those JSON descriptions are
almost merged, distros can start picking this up.

Michal

Michael S. Tsirkin March 10, 2019, 3:56 a.m. UTC | #3

On Sat, Mar 09, 2019 at 01:48:16AM +0100, Laszlo Ersek wrote:
> Repo:   https://github.com/lersek/qemu.git
> Branch: edk2_build
> 
> This series advances the roms/edk2 submodule to the "edk2-stable201903"
> release, and builds and captures platform firmware binaries from that
> release. At this point they are meant to be used by both end-users and
> by Igor's ACPI unit tests in qtest ("make check").
> 
> Previous discussion:
> 
>   [Qemu-devel] bundling edk2 platform firmware images with QEMU
>   http://mid.mail-archive.com/80f0bae3-e79a-bb68-04c4-1c9c684d95b8@redhat.com
>   https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg02601.html
> 
> Note that the series was formatted with "--no-binary" (affecting patch
> #8), therefore it cannot be applied with "git-am". See the remote
> repo/branch reference near the top instead.
> 
> Thanks,
> Laszlo

High time IMO.

Reviewed-by: Michael S. Tsirkin <mst@redhat.com>

Laszlo I suggest you add an entry to MAINTAINERS
and start doing pull requests.

Peter, what do you say? OK with you?


> Laszlo Ersek (10):
>   roms: lift "edk2-funcs.sh" from "tests/uefi-test-tools/build.sh"
>   roms/edk2-funcs.sh: require gcc-4.8+ for building i386 and x86_64
>   tests/uefi-test-tools/build.sh: work around TianoCore#1607
>   roms/edk2: advance to tag edk2-stable201903
>   roms/edk2-funcs.sh: add the qemu_edk2_get_thread_count() function
>   roms/Makefile: replace the $(EFIROM) target with "edk2-basetools"
>   roms: build edk2 firmware binaries and variable store templates
>   pc-bios: add edk2 firmware binaries and variable store templates
>   pc-bios: document the edk2 firmware images; add firmware descriptors
>   Makefile: install the edk2 firmware images and their descriptors
> 
>  Makefile                                       |  17 +-
>  pc-bios/README                                 |  11 +
>  pc-bios/descriptors/50-edk2-i386-secure.json   |  34 +++
>  pc-bios/descriptors/50-edk2-x86_64-secure.json |  35 +++
>  pc-bios/descriptors/60-edk2-aarch64.json       |  31 +++
>  pc-bios/descriptors/60-edk2-arm.json           |  31 +++
>  pc-bios/descriptors/60-edk2-i386.json          |  33 +++
>  pc-bios/descriptors/60-edk2-x86_64.json        |  34 +++
>  pc-bios/edk2-aarch64-code.fd                   | Bin 0 -> 67108864 bytes
>  pc-bios/edk2-arm-code.fd                       | Bin 0 -> 67108864 bytes
>  pc-bios/edk2-arm-vars.fd                       | Bin 0 -> 67108864 bytes
>  pc-bios/edk2-i386-code.fd                      | Bin 0 -> 3653632 bytes
>  pc-bios/edk2-i386-secure-code.fd               | Bin 0 -> 3653632 bytes
>  pc-bios/edk2-i386-vars.fd                      | Bin 0 -> 540672 bytes
>  pc-bios/edk2-licenses.txt                      | 209 ++++++++++++++++
>  pc-bios/edk2-x86_64-code.fd                    | Bin 0 -> 3653632 bytes
>  pc-bios/edk2-x86_64-secure-code.fd             | Bin 0 -> 3653632 bytes
>  roms/Makefile                                  |   9 +-
>  roms/Makefile.edk2                             | 138 +++++++++++
>  roms/edk2                                      |   2 +-
>  roms/edk2-build.sh                             |  55 +++++
>  roms/edk2-funcs.sh                             | 253 ++++++++++++++++++++
>  tests/uefi-test-tools/build.sh                 | 100 +-------
>  23 files changed, 897 insertions(+), 95 deletions(-)
>  create mode 100644 pc-bios/descriptors/50-edk2-i386-secure.json
>  create mode 100644 pc-bios/descriptors/50-edk2-x86_64-secure.json
>  create mode 100644 pc-bios/descriptors/60-edk2-aarch64.json
>  create mode 100644 pc-bios/descriptors/60-edk2-arm.json
>  create mode 100644 pc-bios/descriptors/60-edk2-i386.json
>  create mode 100644 pc-bios/descriptors/60-edk2-x86_64.json
>  create mode 100644 pc-bios/edk2-aarch64-code.fd
>  create mode 100644 pc-bios/edk2-arm-code.fd
>  create mode 100644 pc-bios/edk2-arm-vars.fd
>  create mode 100644 pc-bios/edk2-i386-code.fd
>  create mode 100644 pc-bios/edk2-i386-secure-code.fd
>  create mode 100644 pc-bios/edk2-i386-vars.fd
>  create mode 100644 pc-bios/edk2-licenses.txt
>  create mode 100644 pc-bios/edk2-x86_64-code.fd
>  create mode 100644 pc-bios/edk2-x86_64-secure-code.fd
>  create mode 100644 roms/Makefile.edk2
>  create mode 100755 roms/edk2-build.sh
>  create mode 100644 roms/edk2-funcs.sh
> 
> -- 
> 2.19.1.3.g30247aa5d201

Philippe Mathieu-Daudé March 10, 2019, 11:21 a.m. UTC | #4

On 3/10/19 4:56 AM, Michael S. Tsirkin wrote:
> On Sat, Mar 09, 2019 at 01:48:16AM +0100, Laszlo Ersek wrote:
>> Repo:   https://github.com/lersek/qemu.git
>> Branch: edk2_build
>>
>> This series advances the roms/edk2 submodule to the "edk2-stable201903"
>> release, and builds and captures platform firmware binaries from that
>> release. At this point they are meant to be used by both end-users and
>> by Igor's ACPI unit tests in qtest ("make check").
>>
>> Previous discussion:
>>
>>   [Qemu-devel] bundling edk2 platform firmware images with QEMU
>>   http://mid.mail-archive.com/80f0bae3-e79a-bb68-04c4-1c9c684d95b8@redhat.com
>>   https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg02601.html

There David raised a concern about "[adding] ~206 MB of binaries to the
pc-bios directory". I'm also worried.

GitHub kindly suggest to use git-lfs. It is an extra dependency I'd
rather strongly avoid (because we support a wide range of host OS, each
using a wide types of filesystems).

What about storing those binaries on a file server (http/ftp) altogether
with a file containing its hashed digest (SHA1/SHA256)? Then we already
have all the required tools to fetch and verify those blob roms with the
build system.
Or we could store the hashes in the QEMU repository too.

>> Note that the series was formatted with "--no-binary" (affecting patch
>> #8), therefore it cannot be applied with "git-am". See the remote
>> repo/branch reference near the top instead.
>>
>> Thanks,
>> Laszlo
> 
> High time IMO.

:)

> Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
> 
> Laszlo I suggest you add an entry to MAINTAINERS
> and start doing pull requests.

This is the entry I added here:
https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg02967.html

> 
> Peter, what do you say? OK with you?

Since this series doesn't change the QEMU binaries built, it looks OK to
me to merge it past soft freeze (as we do we tests/CI), this way it get
merged with the final EDK2 release tag.
Else we can merge it next week, and update the EDK2 submodule tag
previous QEMU release.

>> Laszlo Ersek (10):
>>   roms: lift "edk2-funcs.sh" from "tests/uefi-test-tools/build.sh"
>>   roms/edk2-funcs.sh: require gcc-4.8+ for building i386 and x86_64
>>   tests/uefi-test-tools/build.sh: work around TianoCore#1607
>>   roms/edk2: advance to tag edk2-stable201903
>>   roms/edk2-funcs.sh: add the qemu_edk2_get_thread_count() function
>>   roms/Makefile: replace the $(EFIROM) target with "edk2-basetools"
>>   roms: build edk2 firmware binaries and variable store templates
>>   pc-bios: add edk2 firmware binaries and variable store templates
>>   pc-bios: document the edk2 firmware images; add firmware descriptors
>>   Makefile: install the edk2 firmware images and their descriptors
>>
>>  Makefile                                       |  17 +-
>>  pc-bios/README                                 |  11 +
>>  pc-bios/descriptors/50-edk2-i386-secure.json   |  34 +++
>>  pc-bios/descriptors/50-edk2-x86_64-secure.json |  35 +++
>>  pc-bios/descriptors/60-edk2-aarch64.json       |  31 +++
>>  pc-bios/descriptors/60-edk2-arm.json           |  31 +++
>>  pc-bios/descriptors/60-edk2-i386.json          |  33 +++
>>  pc-bios/descriptors/60-edk2-x86_64.json        |  34 +++
>>  pc-bios/edk2-aarch64-code.fd                   | Bin 0 -> 67108864 bytes
>>  pc-bios/edk2-arm-code.fd                       | Bin 0 -> 67108864 bytes
>>  pc-bios/edk2-arm-vars.fd                       | Bin 0 -> 67108864 bytes
>>  pc-bios/edk2-i386-code.fd                      | Bin 0 -> 3653632 bytes
>>  pc-bios/edk2-i386-secure-code.fd               | Bin 0 -> 3653632 bytes
>>  pc-bios/edk2-i386-vars.fd                      | Bin 0 -> 540672 bytes
>>  pc-bios/edk2-licenses.txt                      | 209 ++++++++++++++++
>>  pc-bios/edk2-x86_64-code.fd                    | Bin 0 -> 3653632 bytes
>>  pc-bios/edk2-x86_64-secure-code.fd             | Bin 0 -> 3653632 bytes
>>  roms/Makefile                                  |   9 +-
>>  roms/Makefile.edk2                             | 138 +++++++++++
>>  roms/edk2                                      |   2 +-
>>  roms/edk2-build.sh                             |  55 +++++
>>  roms/edk2-funcs.sh                             | 253 ++++++++++++++++++++
>>  tests/uefi-test-tools/build.sh                 | 100 +-------
>>  23 files changed, 897 insertions(+), 95 deletions(-)
>>  create mode 100644 pc-bios/descriptors/50-edk2-i386-secure.json
>>  create mode 100644 pc-bios/descriptors/50-edk2-x86_64-secure.json
>>  create mode 100644 pc-bios/descriptors/60-edk2-aarch64.json
>>  create mode 100644 pc-bios/descriptors/60-edk2-arm.json
>>  create mode 100644 pc-bios/descriptors/60-edk2-i386.json
>>  create mode 100644 pc-bios/descriptors/60-edk2-x86_64.json
>>  create mode 100644 pc-bios/edk2-aarch64-code.fd
>>  create mode 100644 pc-bios/edk2-arm-code.fd
>>  create mode 100644 pc-bios/edk2-arm-vars.fd
>>  create mode 100644 pc-bios/edk2-i386-code.fd
>>  create mode 100644 pc-bios/edk2-i386-secure-code.fd
>>  create mode 100644 pc-bios/edk2-i386-vars.fd
>>  create mode 100644 pc-bios/edk2-licenses.txt
>>  create mode 100644 pc-bios/edk2-x86_64-code.fd
>>  create mode 100644 pc-bios/edk2-x86_64-secure-code.fd
>>  create mode 100644 roms/Makefile.edk2
>>  create mode 100755 roms/edk2-build.sh
>>  create mode 100644 roms/edk2-funcs.sh
>>
>> -- 
>> 2.19.1.3.g30247aa5d201

Daniel P. Berrangé March 11, 2019, 10:23 a.m. UTC | #5

On Sat, Mar 09, 2019 at 02:10:17PM +0100, Michal Prívozník wrote:
> On 3/9/19 1:48 AM, Laszlo Ersek wrote:
> > Repo:   https://github.com/lersek/qemu.git
> > Branch: edk2_build
> > 
> > This series advances the roms/edk2 submodule to the "edk2-stable201903"
> > release, and builds and captures platform firmware binaries from that
> > release. At this point they are meant to be used by both end-users and
> > by Igor's ACPI unit tests in qtest ("make check").
> > 
> > Previous discussion:
> > 
> >   [Qemu-devel] bundling edk2 platform firmware images with QEMU
> >   http://mid.mail-archive.com/80f0bae3-e79a-bb68-04c4-1c9c684d95b8@redhat.com
> >   https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg02601.html
> > 
> > Note that the series was formatted with "--no-binary" (affecting patch
> > #8), therefore it cannot be applied with "git-am". See the remote
> > repo/branch reference near the top instead.
> > 
> > Thanks,
> > Laszlo
> > 
> > Laszlo Ersek (10):
> >   roms: lift "edk2-funcs.sh" from "tests/uefi-test-tools/build.sh"
> >   roms/edk2-funcs.sh: require gcc-4.8+ for building i386 and x86_64
> >   tests/uefi-test-tools/build.sh: work around TianoCore#1607
> >   roms/edk2: advance to tag edk2-stable201903
> >   roms/edk2-funcs.sh: add the qemu_edk2_get_thread_count() function
> >   roms/Makefile: replace the $(EFIROM) target with "edk2-basetools"
> >   roms: build edk2 firmware binaries and variable store templates
> >   pc-bios: add edk2 firmware binaries and variable store templates
> >   pc-bios: document the edk2 firmware images; add firmware descriptors
> >   Makefile: install the edk2 firmware images and their descriptors
> > 
> >  Makefile                                       |  17 +-
> >  pc-bios/README                                 |  11 +
> >  pc-bios/descriptors/50-edk2-i386-secure.json   |  34 +++
> >  pc-bios/descriptors/50-edk2-x86_64-secure.json |  35 +++
> >  pc-bios/descriptors/60-edk2-aarch64.json       |  31 +++
> >  pc-bios/descriptors/60-edk2-arm.json           |  31 +++
> >  pc-bios/descriptors/60-edk2-i386.json          |  33 +++
> >  pc-bios/descriptors/60-edk2-x86_64.json        |  34 +++
> >  pc-bios/edk2-aarch64-code.fd                   | Bin 0 -> 67108864 bytes
> >  pc-bios/edk2-arm-code.fd                       | Bin 0 -> 67108864 bytes
> >  pc-bios/edk2-arm-vars.fd                       | Bin 0 -> 67108864 bytes
> >  pc-bios/edk2-i386-code.fd                      | Bin 0 -> 3653632 bytes
> >  pc-bios/edk2-i386-secure-code.fd               | Bin 0 -> 3653632 bytes
> >  pc-bios/edk2-i386-vars.fd                      | Bin 0 -> 540672 bytes
> >  pc-bios/edk2-licenses.txt                      | 209 ++++++++++++++++
> >  pc-bios/edk2-x86_64-code.fd                    | Bin 0 -> 3653632 bytes
> >  pc-bios/edk2-x86_64-secure-code.fd             | Bin 0 -> 3653632 bytes
> >  roms/Makefile                                  |   9 +-
> >  roms/Makefile.edk2                             | 138 +++++++++++
> >  roms/edk2                                      |   2 +-
> >  roms/edk2-build.sh                             |  55 +++++
> >  roms/edk2-funcs.sh                             | 253 ++++++++++++++++++++
> >  tests/uefi-test-tools/build.sh                 | 100 +-------
> >  23 files changed, 897 insertions(+), 95 deletions(-)
> >  create mode 100644 pc-bios/descriptors/50-edk2-i386-secure.json
> >  create mode 100644 pc-bios/descriptors/50-edk2-x86_64-secure.json
> >  create mode 100644 pc-bios/descriptors/60-edk2-aarch64.json
> >  create mode 100644 pc-bios/descriptors/60-edk2-arm.json
> >  create mode 100644 pc-bios/descriptors/60-edk2-i386.json
> >  create mode 100644 pc-bios/descriptors/60-edk2-x86_64.json
> >  create mode 100644 pc-bios/edk2-aarch64-code.fd
> >  create mode 100644 pc-bios/edk2-arm-code.fd
> >  create mode 100644 pc-bios/edk2-arm-vars.fd
> >  create mode 100644 pc-bios/edk2-i386-code.fd
> >  create mode 100644 pc-bios/edk2-i386-secure-code.fd
> >  create mode 100644 pc-bios/edk2-i386-vars.fd
> >  create mode 100644 pc-bios/edk2-licenses.txt
> >  create mode 100644 pc-bios/edk2-x86_64-code.fd
> >  create mode 100644 pc-bios/edk2-x86_64-secure-code.fd
> >  create mode 100644 roms/Makefile.edk2
> >  create mode 100755 roms/edk2-build.sh
> >  create mode 100644 roms/edk2-funcs.sh
> > 
> 
> Unsure whether my ACK is worth anything on this list, but you have it.
>
> Reviewed-by: Michal Privoznik <mprivozn@redhat.com>

Anyone on the QEMU mailing list is welcome, and indeed encouraged to
review patches & add R-b acks if they feel comfortable with the code
they are looking at. As with most projects, QEMU suffers from a
not having enough reviewers to look at the volume of patches sent....

Regards,
Daniel

Daniel P. Berrangé March 11, 2019, 10:28 a.m. UTC | #6

On Sat, Mar 09, 2019 at 02:20:06AM +0100, Philippe Mathieu-Daudé wrote:
> On 3/9/19 1:48 AM, Laszlo Ersek wrote:
> > Repo:   https://github.com/lersek/qemu.git
> > Branch: edk2_build
> > 
> > This series advances the roms/edk2 submodule to the "edk2-stable201903"
> > release, and builds and captures platform firmware binaries from that
> > release. At this point they are meant to be used by both end-users and
> > by Igor's ACPI unit tests in qtest ("make check").
> > 
> > Previous discussion:
> > 
> >   [Qemu-devel] bundling edk2 platform firmware images with QEMU
> >   http://mid.mail-archive.com/80f0bae3-e79a-bb68-04c4-1c9c684d95b8@redhat.com
> >   https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg02601.html
> > 
> > Note that the series was formatted with "--no-binary" (affecting patch
> > #8), therefore it cannot be applied with "git-am". See the remote
> > repo/branch reference near the top instead.
> > 
> > Thanks,
> > Laszlo
> > 
> > Laszlo Ersek (10):
> >   roms: lift "edk2-funcs.sh" from "tests/uefi-test-tools/build.sh"
> >   roms/edk2-funcs.sh: require gcc-4.8+ for building i386 and x86_64
> >   tests/uefi-test-tools/build.sh: work around TianoCore#1607
> >   roms/edk2: advance to tag edk2-stable201903
> >   roms/edk2-funcs.sh: add the qemu_edk2_get_thread_count() function
> >   roms/Makefile: replace the $(EFIROM) target with "edk2-basetools"
> >   roms: build edk2 firmware binaries and variable store templates
> >   pc-bios: add edk2 firmware binaries and variable store templates
> >   pc-bios: document the edk2 firmware images; add firmware descriptors
> >   Makefile: install the edk2 firmware images and their descriptors
> > 
> >  Makefile                                       |  17 +-
> >  pc-bios/README                                 |  11 +
> >  pc-bios/descriptors/50-edk2-i386-secure.json   |  34 +++
> >  pc-bios/descriptors/50-edk2-x86_64-secure.json |  35 +++
> >  pc-bios/descriptors/60-edk2-aarch64.json       |  31 +++
> >  pc-bios/descriptors/60-edk2-arm.json           |  31 +++
> >  pc-bios/descriptors/60-edk2-i386.json          |  33 +++
> >  pc-bios/descriptors/60-edk2-x86_64.json        |  34 +++
> >  pc-bios/edk2-aarch64-code.fd                   | Bin 0 -> 67108864 bytes
> >  pc-bios/edk2-arm-code.fd                       | Bin 0 -> 67108864 bytes
> >  pc-bios/edk2-arm-vars.fd                       | Bin 0 -> 67108864 bytes
> 
> GitHub moans here:
> 
> remote: warning: GH001: Large files detected. You may want to try Git
> Large File Storage - https://git-lfs.github.com.
> remote: warning: See http://git.io/iEPt8g for more information.
> remote: warning: File pc-bios/edk2-arm-vars.fd is 64.00 MB; this is
> larger than GitHub's recommended maximum file size of 50.00 MB
> remote: warning: File pc-bios/edk2-arm-code.fd is 64.00 MB; this is
> larger than GitHub's recommended maximum file size of 50.00 MB
> remote: warning: File pc-bios/edk2-aarch64-code.fd is 64.00 MB; this is
> larger than GitHub's recommended maximum file size of 50.00 MB

I wonder if this is a such that github isn't handling sparse files
well, or if they just blindly do this check before they look at the
actual required storage for the files.

Regards,
Daniel

Daniel P. Berrangé March 11, 2019, 10:35 a.m. UTC | #7

On Sun, Mar 10, 2019 at 12:21:55PM +0100, Philippe Mathieu-Daudé wrote:
> On 3/10/19 4:56 AM, Michael S. Tsirkin wrote:
> > On Sat, Mar 09, 2019 at 01:48:16AM +0100, Laszlo Ersek wrote:
> >> Repo:   https://github.com/lersek/qemu.git
> >> Branch: edk2_build
> >>
> >> This series advances the roms/edk2 submodule to the "edk2-stable201903"
> >> release, and builds and captures platform firmware binaries from that
> >> release. At this point they are meant to be used by both end-users and
> >> by Igor's ACPI unit tests in qtest ("make check").
> >>
> >> Previous discussion:
> >>
> >>   [Qemu-devel] bundling edk2 platform firmware images with QEMU
> >>   http://mid.mail-archive.com/80f0bae3-e79a-bb68-04c4-1c9c684d95b8@redhat.com
> >>   https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg02601.html
> 
> There David raised a concern about "[adding] ~206 MB of binaries to the
> pc-bios directory". I'm also worried.
> 
> GitHub kindly suggest to use git-lfs. It is an extra dependency I'd
> rather strongly avoid (because we support a wide range of host OS, each
> using a wide types of filesystems).
> 
> What about storing those binaries on a file server (http/ftp) altogether
> with a file containing its hashed digest (SHA1/SHA256)? Then we already
> have all the required tools to fetch and verify those blob roms with the
> build system.
> Or we could store the hashes in the QEMU repository too.

A simple approach would simply be to 'xz' compress all the edk*.fd
files but still store them in git. They're already opaque binary
files, so replacing one binary format with another binary format
is no big deal IMHO

$ ls -alsh edk2-*
1.1M -rw-rw-r--. 1 berrange berrange 1.1M Mar 11 10:29 edk2-aarch64-code.fd.xz
2.1M -rw-rw-r--. 1 berrange berrange  64M Mar 11 10:29 edk2-arm-code.fd
772K -rw-rw-r--. 1 berrange berrange  64M Mar 11 10:29 edk2-arm-vars.fd
3.5M -rw-rw-r--. 1 berrange berrange 3.5M Mar 11 10:29 edk2-i386-code.fd
3.5M -rw-rw-r--. 1 berrange berrange 3.5M Mar 11 10:29 edk2-i386-secure-code.fd
528K -rw-rw-r--. 1 berrange berrange 528K Mar 11 10:29 edk2-i386-vars.fd
 12K -rw-rw-r--. 1 berrange berrange  11K Mar 11 10:29 edk2-licenses.txt
3.5M -rw-rw-r--. 1 berrange berrange 3.5M Mar 11 10:29 edk2-x86_64-code.fd
3.5M -rw-rw-r--. 1 berrange berrange 3.5M Mar 11 10:29 edk2-x86_64-secure-code.fd

This gives a 50% disk space saving over the sparse size:

$ ls -alsh edk2-*
1.1M -rw-rw-r--. 1 berrange berrange 1.1M Mar 11 10:29 edk2-aarch64-code.fd.xz
1.1M -rw-rw-r--. 1 berrange berrange 1.1M Mar 11 10:29 edk2-arm-code.fd.xz
 12K -rw-rw-r--. 1 berrange berrange 9.8K Mar 11 10:29 edk2-arm-vars.fd.xz
1.6M -rw-rw-r--. 1 berrange berrange 1.6M Mar 11 10:29 edk2-i386-code.fd.xz
1.8M -rw-rw-r--. 1 berrange berrange 1.8M Mar 11 10:29 edk2-i386-secure-code.fd.xz
4.0K -rw-rw-r--. 1 berrange berrange  320 Mar 11 10:29 edk2-i386-vars.fd.xz
 12K -rw-rw-r--. 1 berrange berrange  11K Mar 11 10:29 edk2-licenses.txt
1.6M -rw-rw-r--. 1 berrange berrange 1.6M Mar 11 10:29 edk2-x86_64-code.fd.xz
1.9M -rw-rw-r--. 1 berrange berrange 1.9M Mar 11 10:29 edk2-x86_64-secure-code.fd.xz

So about 9 MB compressed, instead of 20MB for the uncompressed sparse
files, which is on a par with the existing ROM sizes.

We would then need a "make" rule that runs  unxz to "build" the firmware
files. Wouldn't need to be more complicated that this:

   edk2-%.fd: edk2-%.fd.xz
         unzx -c $< > $@

   CLEANFILES += $(wildcard edk2*.fd)

Regards,
Daniel

Michael S. Tsirkin March 11, 2019, 12:57 p.m. UTC | #8

On Mon, Mar 11, 2019 at 10:28:01AM +0000, Daniel P. Berrangé wrote:
> On Sat, Mar 09, 2019 at 02:20:06AM +0100, Philippe Mathieu-Daudé wrote:
> > On 3/9/19 1:48 AM, Laszlo Ersek wrote:
> > > Repo:   https://github.com/lersek/qemu.git
> > > Branch: edk2_build
> > > 
> > > This series advances the roms/edk2 submodule to the "edk2-stable201903"
> > > release, and builds and captures platform firmware binaries from that
> > > release. At this point they are meant to be used by both end-users and
> > > by Igor's ACPI unit tests in qtest ("make check").
> > > 
> > > Previous discussion:
> > > 
> > >   [Qemu-devel] bundling edk2 platform firmware images with QEMU
> > >   http://mid.mail-archive.com/80f0bae3-e79a-bb68-04c4-1c9c684d95b8@redhat.com
> > >   https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg02601.html
> > > 
> > > Note that the series was formatted with "--no-binary" (affecting patch
> > > #8), therefore it cannot be applied with "git-am". See the remote
> > > repo/branch reference near the top instead.
> > > 
> > > Thanks,
> > > Laszlo
> > > 
> > > Laszlo Ersek (10):
> > >   roms: lift "edk2-funcs.sh" from "tests/uefi-test-tools/build.sh"
> > >   roms/edk2-funcs.sh: require gcc-4.8+ for building i386 and x86_64
> > >   tests/uefi-test-tools/build.sh: work around TianoCore#1607
> > >   roms/edk2: advance to tag edk2-stable201903
> > >   roms/edk2-funcs.sh: add the qemu_edk2_get_thread_count() function
> > >   roms/Makefile: replace the $(EFIROM) target with "edk2-basetools"
> > >   roms: build edk2 firmware binaries and variable store templates
> > >   pc-bios: add edk2 firmware binaries and variable store templates
> > >   pc-bios: document the edk2 firmware images; add firmware descriptors
> > >   Makefile: install the edk2 firmware images and their descriptors
> > > 
> > >  Makefile                                       |  17 +-
> > >  pc-bios/README                                 |  11 +
> > >  pc-bios/descriptors/50-edk2-i386-secure.json   |  34 +++
> > >  pc-bios/descriptors/50-edk2-x86_64-secure.json |  35 +++
> > >  pc-bios/descriptors/60-edk2-aarch64.json       |  31 +++
> > >  pc-bios/descriptors/60-edk2-arm.json           |  31 +++
> > >  pc-bios/descriptors/60-edk2-i386.json          |  33 +++
> > >  pc-bios/descriptors/60-edk2-x86_64.json        |  34 +++
> > >  pc-bios/edk2-aarch64-code.fd                   | Bin 0 -> 67108864 bytes
> > >  pc-bios/edk2-arm-code.fd                       | Bin 0 -> 67108864 bytes
> > >  pc-bios/edk2-arm-vars.fd                       | Bin 0 -> 67108864 bytes
> > 
> > GitHub moans here:
> > 
> > remote: warning: GH001: Large files detected. You may want to try Git
> > Large File Storage - https://git-lfs.github.com.
> > remote: warning: See http://git.io/iEPt8g for more information.
> > remote: warning: File pc-bios/edk2-arm-vars.fd is 64.00 MB; this is
> > larger than GitHub's recommended maximum file size of 50.00 MB
> > remote: warning: File pc-bios/edk2-arm-code.fd is 64.00 MB; this is
> > larger than GitHub's recommended maximum file size of 50.00 MB
> > remote: warning: File pc-bios/edk2-aarch64-code.fd is 64.00 MB; this is
> > larger than GitHub's recommended maximum file size of 50.00 MB
> 
> I wonder if this is a such that github isn't handling sparse files
> well, or if they just blindly do this check before they look at the
> actual required storage for the files.
> 
> Regards,
> Daniel


Right. But really: can we keep these around compressed?
People's disk space isn't completely free either
(even though it's getting cheaper).

> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Daniel P. Berrangé March 11, 2019, 1 p.m. UTC | #9

On Mon, Mar 11, 2019 at 08:57:04AM -0400, Michael S. Tsirkin wrote:
> On Mon, Mar 11, 2019 at 10:28:01AM +0000, Daniel P. Berrangé wrote:
> > On Sat, Mar 09, 2019 at 02:20:06AM +0100, Philippe Mathieu-Daudé wrote:
> > > On 3/9/19 1:48 AM, Laszlo Ersek wrote:
> > > > Repo:   https://github.com/lersek/qemu.git
> > > > Branch: edk2_build
> > > > 
> > > > This series advances the roms/edk2 submodule to the "edk2-stable201903"
> > > > release, and builds and captures platform firmware binaries from that
> > > > release. At this point they are meant to be used by both end-users and
> > > > by Igor's ACPI unit tests in qtest ("make check").
> > > > 
> > > > Previous discussion:
> > > > 
> > > >   [Qemu-devel] bundling edk2 platform firmware images with QEMU
> > > >   http://mid.mail-archive.com/80f0bae3-e79a-bb68-04c4-1c9c684d95b8@redhat.com
> > > >   https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg02601.html
> > > > 
> > > > Note that the series was formatted with "--no-binary" (affecting patch
> > > > #8), therefore it cannot be applied with "git-am". See the remote
> > > > repo/branch reference near the top instead.
> > > > 
> > > > Thanks,
> > > > Laszlo
> > > > 
> > > > Laszlo Ersek (10):
> > > >   roms: lift "edk2-funcs.sh" from "tests/uefi-test-tools/build.sh"
> > > >   roms/edk2-funcs.sh: require gcc-4.8+ for building i386 and x86_64
> > > >   tests/uefi-test-tools/build.sh: work around TianoCore#1607
> > > >   roms/edk2: advance to tag edk2-stable201903
> > > >   roms/edk2-funcs.sh: add the qemu_edk2_get_thread_count() function
> > > >   roms/Makefile: replace the $(EFIROM) target with "edk2-basetools"
> > > >   roms: build edk2 firmware binaries and variable store templates
> > > >   pc-bios: add edk2 firmware binaries and variable store templates
> > > >   pc-bios: document the edk2 firmware images; add firmware descriptors
> > > >   Makefile: install the edk2 firmware images and their descriptors
> > > > 
> > > >  Makefile                                       |  17 +-
> > > >  pc-bios/README                                 |  11 +
> > > >  pc-bios/descriptors/50-edk2-i386-secure.json   |  34 +++
> > > >  pc-bios/descriptors/50-edk2-x86_64-secure.json |  35 +++
> > > >  pc-bios/descriptors/60-edk2-aarch64.json       |  31 +++
> > > >  pc-bios/descriptors/60-edk2-arm.json           |  31 +++
> > > >  pc-bios/descriptors/60-edk2-i386.json          |  33 +++
> > > >  pc-bios/descriptors/60-edk2-x86_64.json        |  34 +++
> > > >  pc-bios/edk2-aarch64-code.fd                   | Bin 0 -> 67108864 bytes
> > > >  pc-bios/edk2-arm-code.fd                       | Bin 0 -> 67108864 bytes
> > > >  pc-bios/edk2-arm-vars.fd                       | Bin 0 -> 67108864 bytes
> > > 
> > > GitHub moans here:
> > > 
> > > remote: warning: GH001: Large files detected. You may want to try Git
> > > Large File Storage - https://git-lfs.github.com.
> > > remote: warning: See http://git.io/iEPt8g for more information.
> > > remote: warning: File pc-bios/edk2-arm-vars.fd is 64.00 MB; this is
> > > larger than GitHub's recommended maximum file size of 50.00 MB
> > > remote: warning: File pc-bios/edk2-arm-code.fd is 64.00 MB; this is
> > > larger than GitHub's recommended maximum file size of 50.00 MB
> > > remote: warning: File pc-bios/edk2-aarch64-code.fd is 64.00 MB; this is
> > > larger than GitHub's recommended maximum file size of 50.00 MB
> > 
> > I wonder if this is a such that github isn't handling sparse files
> > well, or if they just blindly do this check before they look at the
> > actual required storage for the files.
> > 
> > Regards,
> > Daniel
> 
> 
> Right. But really: can we keep these around compressed?

I think it is viable for us to xz compress the images that we store in
git & just let make "build"  the uncompressed images when needed.

Regards,
Daniel

Michael S. Tsirkin March 11, 2019, 1:04 p.m. UTC | #10

On Mon, Mar 11, 2019 at 01:00:00PM +0000, Daniel P. Berrangé wrote:
> On Mon, Mar 11, 2019 at 08:57:04AM -0400, Michael S. Tsirkin wrote:
> > On Mon, Mar 11, 2019 at 10:28:01AM +0000, Daniel P. Berrangé wrote:
> > > On Sat, Mar 09, 2019 at 02:20:06AM +0100, Philippe Mathieu-Daudé wrote:
> > > > On 3/9/19 1:48 AM, Laszlo Ersek wrote:
> > > > > Repo:   https://github.com/lersek/qemu.git
> > > > > Branch: edk2_build
> > > > > 
> > > > > This series advances the roms/edk2 submodule to the "edk2-stable201903"
> > > > > release, and builds and captures platform firmware binaries from that
> > > > > release. At this point they are meant to be used by both end-users and
> > > > > by Igor's ACPI unit tests in qtest ("make check").
> > > > > 
> > > > > Previous discussion:
> > > > > 
> > > > >   [Qemu-devel] bundling edk2 platform firmware images with QEMU
> > > > >   http://mid.mail-archive.com/80f0bae3-e79a-bb68-04c4-1c9c684d95b8@redhat.com
> > > > >   https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg02601.html
> > > > > 
> > > > > Note that the series was formatted with "--no-binary" (affecting patch
> > > > > #8), therefore it cannot be applied with "git-am". See the remote
> > > > > repo/branch reference near the top instead.
> > > > > 
> > > > > Thanks,
> > > > > Laszlo
> > > > > 
> > > > > Laszlo Ersek (10):
> > > > >   roms: lift "edk2-funcs.sh" from "tests/uefi-test-tools/build.sh"
> > > > >   roms/edk2-funcs.sh: require gcc-4.8+ for building i386 and x86_64
> > > > >   tests/uefi-test-tools/build.sh: work around TianoCore#1607
> > > > >   roms/edk2: advance to tag edk2-stable201903
> > > > >   roms/edk2-funcs.sh: add the qemu_edk2_get_thread_count() function
> > > > >   roms/Makefile: replace the $(EFIROM) target with "edk2-basetools"
> > > > >   roms: build edk2 firmware binaries and variable store templates
> > > > >   pc-bios: add edk2 firmware binaries and variable store templates
> > > > >   pc-bios: document the edk2 firmware images; add firmware descriptors
> > > > >   Makefile: install the edk2 firmware images and their descriptors
> > > > > 
> > > > >  Makefile                                       |  17 +-
> > > > >  pc-bios/README                                 |  11 +
> > > > >  pc-bios/descriptors/50-edk2-i386-secure.json   |  34 +++
> > > > >  pc-bios/descriptors/50-edk2-x86_64-secure.json |  35 +++
> > > > >  pc-bios/descriptors/60-edk2-aarch64.json       |  31 +++
> > > > >  pc-bios/descriptors/60-edk2-arm.json           |  31 +++
> > > > >  pc-bios/descriptors/60-edk2-i386.json          |  33 +++
> > > > >  pc-bios/descriptors/60-edk2-x86_64.json        |  34 +++
> > > > >  pc-bios/edk2-aarch64-code.fd                   | Bin 0 -> 67108864 bytes
> > > > >  pc-bios/edk2-arm-code.fd                       | Bin 0 -> 67108864 bytes
> > > > >  pc-bios/edk2-arm-vars.fd                       | Bin 0 -> 67108864 bytes
> > > > 
> > > > GitHub moans here:
> > > > 
> > > > remote: warning: GH001: Large files detected. You may want to try Git
> > > > Large File Storage - https://git-lfs.github.com.
> > > > remote: warning: See http://git.io/iEPt8g for more information.
> > > > remote: warning: File pc-bios/edk2-arm-vars.fd is 64.00 MB; this is
> > > > larger than GitHub's recommended maximum file size of 50.00 MB
> > > > remote: warning: File pc-bios/edk2-arm-code.fd is 64.00 MB; this is
> > > > larger than GitHub's recommended maximum file size of 50.00 MB
> > > > remote: warning: File pc-bios/edk2-aarch64-code.fd is 64.00 MB; this is
> > > > larger than GitHub's recommended maximum file size of 50.00 MB
> > > 
> > > I wonder if this is a such that github isn't handling sparse files
> > > well, or if they just blindly do this check before they look at the
> > > actual required storage for the files.
> > > 
> > > Regards,
> > > Daniel
> > 
> > 
> > Right. But really: can we keep these around compressed?
> 
> I think it is viable for us to xz compress the images that we store in
> git & just let make "build"  the uncompressed images when needed.
> 
> Regards,
> Daniel

Right that's the simplest approach. OTOH we do link with zlib already,
so we could support actual compressed firmware too. Not sure it's worth
it.


> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Philippe Mathieu-Daudé March 11, 2019, 1:15 p.m. UTC | #11

On 3/11/19 1:57 PM, Michael S. Tsirkin wrote:
> On Mon, Mar 11, 2019 at 10:28:01AM +0000, Daniel P. Berrangé wrote:
>> On Sat, Mar 09, 2019 at 02:20:06AM +0100, Philippe Mathieu-Daudé wrote:
>>> On 3/9/19 1:48 AM, Laszlo Ersek wrote:
>>>> Repo:   https://github.com/lersek/qemu.git
>>>> Branch: edk2_build
>>>>
>>>> This series advances the roms/edk2 submodule to the "edk2-stable201903"
>>>> release, and builds and captures platform firmware binaries from that
>>>> release. At this point they are meant to be used by both end-users and
>>>> by Igor's ACPI unit tests in qtest ("make check").
>>>>
>>>> Previous discussion:
>>>>
>>>>   [Qemu-devel] bundling edk2 platform firmware images with QEMU
>>>>   http://mid.mail-archive.com/80f0bae3-e79a-bb68-04c4-1c9c684d95b8@redhat.com
>>>>   https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg02601.html
>>>>
>>>> Note that the series was formatted with "--no-binary" (affecting patch
>>>> #8), therefore it cannot be applied with "git-am". See the remote
>>>> repo/branch reference near the top instead.
>>>>
>>>> Thanks,
>>>> Laszlo
>>>>
>>>> Laszlo Ersek (10):
>>>>   roms: lift "edk2-funcs.sh" from "tests/uefi-test-tools/build.sh"
>>>>   roms/edk2-funcs.sh: require gcc-4.8+ for building i386 and x86_64
>>>>   tests/uefi-test-tools/build.sh: work around TianoCore#1607
>>>>   roms/edk2: advance to tag edk2-stable201903
>>>>   roms/edk2-funcs.sh: add the qemu_edk2_get_thread_count() function
>>>>   roms/Makefile: replace the $(EFIROM) target with "edk2-basetools"
>>>>   roms: build edk2 firmware binaries and variable store templates
>>>>   pc-bios: add edk2 firmware binaries and variable store templates
>>>>   pc-bios: document the edk2 firmware images; add firmware descriptors
>>>>   Makefile: install the edk2 firmware images and their descriptors
>>>>
>>>>  Makefile                                       |  17 +-
>>>>  pc-bios/README                                 |  11 +
>>>>  pc-bios/descriptors/50-edk2-i386-secure.json   |  34 +++
>>>>  pc-bios/descriptors/50-edk2-x86_64-secure.json |  35 +++
>>>>  pc-bios/descriptors/60-edk2-aarch64.json       |  31 +++
>>>>  pc-bios/descriptors/60-edk2-arm.json           |  31 +++
>>>>  pc-bios/descriptors/60-edk2-i386.json          |  33 +++
>>>>  pc-bios/descriptors/60-edk2-x86_64.json        |  34 +++
>>>>  pc-bios/edk2-aarch64-code.fd                   | Bin 0 -> 67108864 bytes
>>>>  pc-bios/edk2-arm-code.fd                       | Bin 0 -> 67108864 bytes
>>>>  pc-bios/edk2-arm-vars.fd                       | Bin 0 -> 67108864 bytes
>>>
>>> GitHub moans here:
>>>
>>> remote: warning: GH001: Large files detected. You may want to try Git
>>> Large File Storage - https://git-lfs.github.com.
>>> remote: warning: See http://git.io/iEPt8g for more information.
>>> remote: warning: File pc-bios/edk2-arm-vars.fd is 64.00 MB; this is
>>> larger than GitHub's recommended maximum file size of 50.00 MB
>>> remote: warning: File pc-bios/edk2-arm-code.fd is 64.00 MB; this is
>>> larger than GitHub's recommended maximum file size of 50.00 MB
>>> remote: warning: File pc-bios/edk2-aarch64-code.fd is 64.00 MB; this is
>>> larger than GitHub's recommended maximum file size of 50.00 MB
>>
>> I wonder if this is a such that github isn't handling sparse files
>> well, or if they just blindly do this check before they look at the
>> actual required storage for the files.
>>
>> Regards,
>> Daniel
> 
> 
> Right. But really: can we keep these around compressed?
> People's disk space isn't completely free either
> (even though it's getting cheaper).

And not everybody has access to a decent Internet connection.

Laszlo Ersek March 12, 2019, 2:22 p.m. UTC | #12

On 03/09/19 02:20, Philippe Mathieu-Daudé wrote:
> On 3/9/19 1:48 AM, Laszlo Ersek wrote:
>> Repo:   https://github.com/lersek/qemu.git
>> Branch: edk2_build
>>
>> This series advances the roms/edk2 submodule to the "edk2-stable201903"
>> release, and builds and captures platform firmware binaries from that
>> release. At this point they are meant to be used by both end-users and
>> by Igor's ACPI unit tests in qtest ("make check").
>>
>> Previous discussion:
>>
>>   [Qemu-devel] bundling edk2 platform firmware images with QEMU
>>   http://mid.mail-archive.com/80f0bae3-e79a-bb68-04c4-1c9c684d95b8@redhat.com
>>   https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg02601.html
>>
>> Note that the series was formatted with "--no-binary" (affecting patch
>> #8), therefore it cannot be applied with "git-am". See the remote
>> repo/branch reference near the top instead.
>>
>> Thanks,
>> Laszlo
>>
>> Laszlo Ersek (10):
>>   roms: lift "edk2-funcs.sh" from "tests/uefi-test-tools/build.sh"
>>   roms/edk2-funcs.sh: require gcc-4.8+ for building i386 and x86_64
>>   tests/uefi-test-tools/build.sh: work around TianoCore#1607
>>   roms/edk2: advance to tag edk2-stable201903
>>   roms/edk2-funcs.sh: add the qemu_edk2_get_thread_count() function
>>   roms/Makefile: replace the $(EFIROM) target with "edk2-basetools"
>>   roms: build edk2 firmware binaries and variable store templates
>>   pc-bios: add edk2 firmware binaries and variable store templates
>>   pc-bios: document the edk2 firmware images; add firmware descriptors
>>   Makefile: install the edk2 firmware images and their descriptors
>>
>>  Makefile                                       |  17 +-
>>  pc-bios/README                                 |  11 +
>>  pc-bios/descriptors/50-edk2-i386-secure.json   |  34 +++
>>  pc-bios/descriptors/50-edk2-x86_64-secure.json |  35 +++
>>  pc-bios/descriptors/60-edk2-aarch64.json       |  31 +++
>>  pc-bios/descriptors/60-edk2-arm.json           |  31 +++
>>  pc-bios/descriptors/60-edk2-i386.json          |  33 +++
>>  pc-bios/descriptors/60-edk2-x86_64.json        |  34 +++
>>  pc-bios/edk2-aarch64-code.fd                   | Bin 0 -> 67108864 bytes
>>  pc-bios/edk2-arm-code.fd                       | Bin 0 -> 67108864 bytes
>>  pc-bios/edk2-arm-vars.fd                       | Bin 0 -> 67108864 bytes
> 
> GitHub moans here:
> 
> remote: warning: GH001: Large files detected. You may want to try Git
> Large File Storage - https://git-lfs.github.com.
> remote: warning: See http://git.io/iEPt8g for more information.
> remote: warning: File pc-bios/edk2-arm-vars.fd is 64.00 MB; this is
> larger than GitHub's recommended maximum file size of 50.00 MB
> remote: warning: File pc-bios/edk2-arm-code.fd is 64.00 MB; this is
> larger than GitHub's recommended maximum file size of 50.00 MB
> remote: warning: File pc-bios/edk2-aarch64-code.fd is 64.00 MB; this is
> larger than GitHub's recommended maximum file size of 50.00 MB

Yes. And, it's GitHub's problem. (It moaned also when I pushed the patches.)

As discussed in the earlier thread (please find the root link above),
the actual data transfer size (and the footprint in the compressed git
object store) is 9MB.

Thanks
Laszlo

> 
>>  pc-bios/edk2-i386-code.fd                      | Bin 0 -> 3653632 bytes
>>  pc-bios/edk2-i386-secure-code.fd               | Bin 0 -> 3653632 bytes
>>  pc-bios/edk2-i386-vars.fd                      | Bin 0 -> 540672 bytes
>>  pc-bios/edk2-licenses.txt                      | 209 ++++++++++++++++
>>  pc-bios/edk2-x86_64-code.fd                    | Bin 0 -> 3653632 bytes
>>  pc-bios/edk2-x86_64-secure-code.fd             | Bin 0 -> 3653632 bytes
>>  roms/Makefile                                  |   9 +-
>>  roms/Makefile.edk2                             | 138 +++++++++++
>>  roms/edk2                                      |   2 +-
>>  roms/edk2-build.sh                             |  55 +++++
>>  roms/edk2-funcs.sh                             | 253 ++++++++++++++++++++
>>  tests/uefi-test-tools/build.sh                 | 100 +-------
>>  23 files changed, 897 insertions(+), 95 deletions(-)
>>  create mode 100644 pc-bios/descriptors/50-edk2-i386-secure.json
>>  create mode 100644 pc-bios/descriptors/50-edk2-x86_64-secure.json
>>  create mode 100644 pc-bios/descriptors/60-edk2-aarch64.json
>>  create mode 100644 pc-bios/descriptors/60-edk2-arm.json
>>  create mode 100644 pc-bios/descriptors/60-edk2-i386.json
>>  create mode 100644 pc-bios/descriptors/60-edk2-x86_64.json
>>  create mode 100644 pc-bios/edk2-aarch64-code.fd
>>  create mode 100644 pc-bios/edk2-arm-code.fd
>>  create mode 100644 pc-bios/edk2-arm-vars.fd
>>  create mode 100644 pc-bios/edk2-i386-code.fd
>>  create mode 100644 pc-bios/edk2-i386-secure-code.fd
>>  create mode 100644 pc-bios/edk2-i386-vars.fd
>>  create mode 100644 pc-bios/edk2-licenses.txt
>>  create mode 100644 pc-bios/edk2-x86_64-code.fd
>>  create mode 100644 pc-bios/edk2-x86_64-secure-code.fd
>>  create mode 100644 roms/Makefile.edk2
>>  create mode 100755 roms/edk2-build.sh
>>  create mode 100644 roms/edk2-funcs.sh
>>

Laszlo Ersek March 12, 2019, 2:30 p.m. UTC | #13

On 03/11/19 11:23, Daniel P. Berrangé wrote:
> On Sat, Mar 09, 2019 at 02:10:17PM +0100, Michal Prívozník wrote:
>> On 3/9/19 1:48 AM, Laszlo Ersek wrote:
>>> Repo:   https://github.com/lersek/qemu.git
>>> Branch: edk2_build
>>>
>>> This series advances the roms/edk2 submodule to the "edk2-stable201903"
>>> release, and builds and captures platform firmware binaries from that
>>> release. At this point they are meant to be used by both end-users and
>>> by Igor's ACPI unit tests in qtest ("make check").
>>>
>>> Previous discussion:
>>>
>>>   [Qemu-devel] bundling edk2 platform firmware images with QEMU
>>>   http://mid.mail-archive.com/80f0bae3-e79a-bb68-04c4-1c9c684d95b8@redhat.com
>>>   https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg02601.html
>>>
>>> Note that the series was formatted with "--no-binary" (affecting patch
>>> #8), therefore it cannot be applied with "git-am". See the remote
>>> repo/branch reference near the top instead.
>>>
>>> Thanks,
>>> Laszlo
>>>
>>> Laszlo Ersek (10):
>>>   roms: lift "edk2-funcs.sh" from "tests/uefi-test-tools/build.sh"
>>>   roms/edk2-funcs.sh: require gcc-4.8+ for building i386 and x86_64
>>>   tests/uefi-test-tools/build.sh: work around TianoCore#1607
>>>   roms/edk2: advance to tag edk2-stable201903
>>>   roms/edk2-funcs.sh: add the qemu_edk2_get_thread_count() function
>>>   roms/Makefile: replace the $(EFIROM) target with "edk2-basetools"
>>>   roms: build edk2 firmware binaries and variable store templates
>>>   pc-bios: add edk2 firmware binaries and variable store templates
>>>   pc-bios: document the edk2 firmware images; add firmware descriptors
>>>   Makefile: install the edk2 firmware images and their descriptors
>>>
>>>  Makefile                                       |  17 +-
>>>  pc-bios/README                                 |  11 +
>>>  pc-bios/descriptors/50-edk2-i386-secure.json   |  34 +++
>>>  pc-bios/descriptors/50-edk2-x86_64-secure.json |  35 +++
>>>  pc-bios/descriptors/60-edk2-aarch64.json       |  31 +++
>>>  pc-bios/descriptors/60-edk2-arm.json           |  31 +++
>>>  pc-bios/descriptors/60-edk2-i386.json          |  33 +++
>>>  pc-bios/descriptors/60-edk2-x86_64.json        |  34 +++
>>>  pc-bios/edk2-aarch64-code.fd                   | Bin 0 -> 67108864 bytes
>>>  pc-bios/edk2-arm-code.fd                       | Bin 0 -> 67108864 bytes
>>>  pc-bios/edk2-arm-vars.fd                       | Bin 0 -> 67108864 bytes
>>>  pc-bios/edk2-i386-code.fd                      | Bin 0 -> 3653632 bytes
>>>  pc-bios/edk2-i386-secure-code.fd               | Bin 0 -> 3653632 bytes
>>>  pc-bios/edk2-i386-vars.fd                      | Bin 0 -> 540672 bytes
>>>  pc-bios/edk2-licenses.txt                      | 209 ++++++++++++++++
>>>  pc-bios/edk2-x86_64-code.fd                    | Bin 0 -> 3653632 bytes
>>>  pc-bios/edk2-x86_64-secure-code.fd             | Bin 0 -> 3653632 bytes
>>>  roms/Makefile                                  |   9 +-
>>>  roms/Makefile.edk2                             | 138 +++++++++++
>>>  roms/edk2                                      |   2 +-
>>>  roms/edk2-build.sh                             |  55 +++++
>>>  roms/edk2-funcs.sh                             | 253 ++++++++++++++++++++
>>>  tests/uefi-test-tools/build.sh                 | 100 +-------
>>>  23 files changed, 897 insertions(+), 95 deletions(-)
>>>  create mode 100644 pc-bios/descriptors/50-edk2-i386-secure.json
>>>  create mode 100644 pc-bios/descriptors/50-edk2-x86_64-secure.json
>>>  create mode 100644 pc-bios/descriptors/60-edk2-aarch64.json
>>>  create mode 100644 pc-bios/descriptors/60-edk2-arm.json
>>>  create mode 100644 pc-bios/descriptors/60-edk2-i386.json
>>>  create mode 100644 pc-bios/descriptors/60-edk2-x86_64.json
>>>  create mode 100644 pc-bios/edk2-aarch64-code.fd
>>>  create mode 100644 pc-bios/edk2-arm-code.fd
>>>  create mode 100644 pc-bios/edk2-arm-vars.fd
>>>  create mode 100644 pc-bios/edk2-i386-code.fd
>>>  create mode 100644 pc-bios/edk2-i386-secure-code.fd
>>>  create mode 100644 pc-bios/edk2-i386-vars.fd
>>>  create mode 100644 pc-bios/edk2-licenses.txt
>>>  create mode 100644 pc-bios/edk2-x86_64-code.fd
>>>  create mode 100644 pc-bios/edk2-x86_64-secure-code.fd
>>>  create mode 100644 roms/Makefile.edk2
>>>  create mode 100755 roms/edk2-build.sh
>>>  create mode 100644 roms/edk2-funcs.sh
>>>
>>
>> Unsure whether my ACK is worth anything on this list, but you have it.
>>
>> Reviewed-by: Michal Privoznik <mprivozn@redhat.com>
> 
> Anyone on the QEMU mailing list is welcome, and indeed encouraged to
> review patches & add R-b acks if they feel comfortable with the code
> they are looking at.

Agreed! Thank you, Michal, for looking at the series!

> As with most projects, QEMU suffers from a
> not having enough reviewers to look at the volume of patches sent....

Agreed. I think this is one of the serious problems that FLOSS projects
face today.

Thanks
Laszlo

Laszlo Ersek March 12, 2019, 3:33 p.m. UTC | #14

On 03/10/19 12:21, Philippe Mathieu-Daudé wrote:
> On 3/10/19 4:56 AM, Michael S. Tsirkin wrote:
>> On Sat, Mar 09, 2019 at 01:48:16AM +0100, Laszlo Ersek wrote:
>>> Repo:   https://github.com/lersek/qemu.git
>>> Branch: edk2_build
>>>
>>> This series advances the roms/edk2 submodule to the "edk2-stable201903"
>>> release, and builds and captures platform firmware binaries from that
>>> release. At this point they are meant to be used by both end-users and
>>> by Igor's ACPI unit tests in qtest ("make check").
>>>
>>> Previous discussion:
>>>
>>>   [Qemu-devel] bundling edk2 platform firmware images with QEMU
>>>   http://mid.mail-archive.com/80f0bae3-e79a-bb68-04c4-1c9c684d95b8@redhat.com
>>>   https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg02601.html
> 
> There David raised a concern about "[adding] ~206 MB of binaries to the
> pc-bios directory".

(I think the concern was voiced by Dan first.)

> I'm also worried.

I'm not. The actual data size, both to transfer (push/pull) and to store
in the .git subdirectory, is ~9MB. (As I explained earlier.)

git-checkout *could* in theory eat more space, but, in my testing, it
doesn't; for me, git-checkout punches holes in the checked-out files.
Please see the earlier discussion for that reference too. (Note: I'm on
ext4, so nothing enterprisey like xfs.)

> GitHub kindly suggest to use git-lfs.

Just ignore that -- not because it's another dependency, but because
external storage for these blobs is actually not required.

> It is an extra dependency I'd
> rather strongly avoid (because we support a wide range of host OS, each
> using a wide types of filesystems).
> 
> What about storing those binaries on a file server (http/ftp) altogether
> with a file containing its hashed digest (SHA1/SHA256)? Then we already
> have all the required tools to fetch and verify those blob roms with the
> build system.
> Or we could store the hashes in the QEMU repository too.

Let's not implement "fedpkg sources" within upstream QEMU, unless we
really must.

> 
>>> Note that the series was formatted with "--no-binary" (affecting patch
>>> #8), therefore it cannot be applied with "git-am". See the remote
>>> repo/branch reference near the top instead.
>>>
>>> Thanks,
>>> Laszlo
>>
>> High time IMO.
> 
> :)
> 
>> Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
>>
>> Laszlo I suggest you add an entry to MAINTAINERS
>> and start doing pull requests.

I'm glad to do that (in the hope that updates will be mostly painless,
regarding the build scaffolding).

> 
> This is the entry I added here:
> https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg02967.html

Hmm. That hunk is insufficient (it doesn't cover the files being added
here), but if I add another hunk myself, then we'll have redundancy.

In other words, this is a conflict. How should we resolve it? Should I
add a separate patch at the end, and let Michael drop it or resolve the
conflict, dependent on the order in which the series end up being merged?

> 
>>
>> Peter, what do you say? OK with you?
> 
> Since this series doesn't change the QEMU binaries built, it looks OK to
> me to merge it past soft freeze (as we do we tests/CI), this way it get
> merged with the final EDK2 release tag.
> Else we can merge it next week, and update the EDK2 submodule tag
> previous QEMU release.

I don't understand this paragraph -- the edk2 submodule commit reference
is already the final one I have targeted for this work, namely
"edk2-stable201903". (I had worried edk2 would suffer a delay in
publishing the tag, but ultimately the community managed it just in time.)

Thanks
Laszlo

> 
>>> Laszlo Ersek (10):
>>>   roms: lift "edk2-funcs.sh" from "tests/uefi-test-tools/build.sh"
>>>   roms/edk2-funcs.sh: require gcc-4.8+ for building i386 and x86_64
>>>   tests/uefi-test-tools/build.sh: work around TianoCore#1607
>>>   roms/edk2: advance to tag edk2-stable201903
>>>   roms/edk2-funcs.sh: add the qemu_edk2_get_thread_count() function
>>>   roms/Makefile: replace the $(EFIROM) target with "edk2-basetools"
>>>   roms: build edk2 firmware binaries and variable store templates
>>>   pc-bios: add edk2 firmware binaries and variable store templates
>>>   pc-bios: document the edk2 firmware images; add firmware descriptors
>>>   Makefile: install the edk2 firmware images and their descriptors
>>>
>>>  Makefile                                       |  17 +-
>>>  pc-bios/README                                 |  11 +
>>>  pc-bios/descriptors/50-edk2-i386-secure.json   |  34 +++
>>>  pc-bios/descriptors/50-edk2-x86_64-secure.json |  35 +++
>>>  pc-bios/descriptors/60-edk2-aarch64.json       |  31 +++
>>>  pc-bios/descriptors/60-edk2-arm.json           |  31 +++
>>>  pc-bios/descriptors/60-edk2-i386.json          |  33 +++
>>>  pc-bios/descriptors/60-edk2-x86_64.json        |  34 +++
>>>  pc-bios/edk2-aarch64-code.fd                   | Bin 0 -> 67108864 bytes
>>>  pc-bios/edk2-arm-code.fd                       | Bin 0 -> 67108864 bytes
>>>  pc-bios/edk2-arm-vars.fd                       | Bin 0 -> 67108864 bytes
>>>  pc-bios/edk2-i386-code.fd                      | Bin 0 -> 3653632 bytes
>>>  pc-bios/edk2-i386-secure-code.fd               | Bin 0 -> 3653632 bytes
>>>  pc-bios/edk2-i386-vars.fd                      | Bin 0 -> 540672 bytes
>>>  pc-bios/edk2-licenses.txt                      | 209 ++++++++++++++++
>>>  pc-bios/edk2-x86_64-code.fd                    | Bin 0 -> 3653632 bytes
>>>  pc-bios/edk2-x86_64-secure-code.fd             | Bin 0 -> 3653632 bytes
>>>  roms/Makefile                                  |   9 +-
>>>  roms/Makefile.edk2                             | 138 +++++++++++
>>>  roms/edk2                                      |   2 +-
>>>  roms/edk2-build.sh                             |  55 +++++
>>>  roms/edk2-funcs.sh                             | 253 ++++++++++++++++++++
>>>  tests/uefi-test-tools/build.sh                 | 100 +-------
>>>  23 files changed, 897 insertions(+), 95 deletions(-)
>>>  create mode 100644 pc-bios/descriptors/50-edk2-i386-secure.json
>>>  create mode 100644 pc-bios/descriptors/50-edk2-x86_64-secure.json
>>>  create mode 100644 pc-bios/descriptors/60-edk2-aarch64.json
>>>  create mode 100644 pc-bios/descriptors/60-edk2-arm.json
>>>  create mode 100644 pc-bios/descriptors/60-edk2-i386.json
>>>  create mode 100644 pc-bios/descriptors/60-edk2-x86_64.json
>>>  create mode 100644 pc-bios/edk2-aarch64-code.fd
>>>  create mode 100644 pc-bios/edk2-arm-code.fd
>>>  create mode 100644 pc-bios/edk2-arm-vars.fd
>>>  create mode 100644 pc-bios/edk2-i386-code.fd
>>>  create mode 100644 pc-bios/edk2-i386-secure-code.fd
>>>  create mode 100644 pc-bios/edk2-i386-vars.fd
>>>  create mode 100644 pc-bios/edk2-licenses.txt
>>>  create mode 100644 pc-bios/edk2-x86_64-code.fd
>>>  create mode 100644 pc-bios/edk2-x86_64-secure-code.fd
>>>  create mode 100644 roms/Makefile.edk2
>>>  create mode 100755 roms/edk2-build.sh
>>>  create mode 100644 roms/edk2-funcs.sh
>>>
>>> -- 
>>> 2.19.1.3.g30247aa5d201

Laszlo Ersek March 12, 2019, 3:40 p.m. UTC | #15

On 03/11/19 11:35, Daniel P. Berrangé wrote:
> On Sun, Mar 10, 2019 at 12:21:55PM +0100, Philippe Mathieu-Daudé wrote:
>> On 3/10/19 4:56 AM, Michael S. Tsirkin wrote:
>>> On Sat, Mar 09, 2019 at 01:48:16AM +0100, Laszlo Ersek wrote:
>>>> Repo:   https://github.com/lersek/qemu.git
>>>> Branch: edk2_build
>>>>
>>>> This series advances the roms/edk2 submodule to the "edk2-stable201903"
>>>> release, and builds and captures platform firmware binaries from that
>>>> release. At this point they are meant to be used by both end-users and
>>>> by Igor's ACPI unit tests in qtest ("make check").
>>>>
>>>> Previous discussion:
>>>>
>>>>   [Qemu-devel] bundling edk2 platform firmware images with QEMU
>>>>   http://mid.mail-archive.com/80f0bae3-e79a-bb68-04c4-1c9c684d95b8@redhat.com
>>>>   https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg02601.html
>>
>> There David raised a concern about "[adding] ~206 MB of binaries to the
>> pc-bios directory". I'm also worried.
>>
>> GitHub kindly suggest to use git-lfs. It is an extra dependency I'd
>> rather strongly avoid (because we support a wide range of host OS, each
>> using a wide types of filesystems).
>>
>> What about storing those binaries on a file server (http/ftp) altogether
>> with a file containing its hashed digest (SHA1/SHA256)? Then we already
>> have all the required tools to fetch and verify those blob roms with the
>> build system.
>> Or we could store the hashes in the QEMU repository too.
> 
> A simple approach would simply be to 'xz' compress all the edk*.fd
> files but still store them in git. They're already opaque binary
> files, so replacing one binary format with another binary format
> is no big deal IMHO
> 
> $ ls -alsh edk2-*
> 1.1M -rw-rw-r--. 1 berrange berrange 1.1M Mar 11 10:29 edk2-aarch64-code.fd.xz
> 2.1M -rw-rw-r--. 1 berrange berrange  64M Mar 11 10:29 edk2-arm-code.fd
> 772K -rw-rw-r--. 1 berrange berrange  64M Mar 11 10:29 edk2-arm-vars.fd
> 3.5M -rw-rw-r--. 1 berrange berrange 3.5M Mar 11 10:29 edk2-i386-code.fd
> 3.5M -rw-rw-r--. 1 berrange berrange 3.5M Mar 11 10:29 edk2-i386-secure-code.fd
> 528K -rw-rw-r--. 1 berrange berrange 528K Mar 11 10:29 edk2-i386-vars.fd
>  12K -rw-rw-r--. 1 berrange berrange  11K Mar 11 10:29 edk2-licenses.txt
> 3.5M -rw-rw-r--. 1 berrange berrange 3.5M Mar 11 10:29 edk2-x86_64-code.fd
> 3.5M -rw-rw-r--. 1 berrange berrange 3.5M Mar 11 10:29 edk2-x86_64-secure-code.fd
> 
> This gives a 50% disk space saving over the sparse size:
> 
> $ ls -alsh edk2-*
> 1.1M -rw-rw-r--. 1 berrange berrange 1.1M Mar 11 10:29 edk2-aarch64-code.fd.xz
> 1.1M -rw-rw-r--. 1 berrange berrange 1.1M Mar 11 10:29 edk2-arm-code.fd.xz
>  12K -rw-rw-r--. 1 berrange berrange 9.8K Mar 11 10:29 edk2-arm-vars.fd.xz
> 1.6M -rw-rw-r--. 1 berrange berrange 1.6M Mar 11 10:29 edk2-i386-code.fd.xz
> 1.8M -rw-rw-r--. 1 berrange berrange 1.8M Mar 11 10:29 edk2-i386-secure-code.fd.xz
> 4.0K -rw-rw-r--. 1 berrange berrange  320 Mar 11 10:29 edk2-i386-vars.fd.xz
>  12K -rw-rw-r--. 1 berrange berrange  11K Mar 11 10:29 edk2-licenses.txt
> 1.6M -rw-rw-r--. 1 berrange berrange 1.6M Mar 11 10:29 edk2-x86_64-code.fd.xz
> 1.9M -rw-rw-r--. 1 berrange berrange 1.9M Mar 11 10:29 edk2-x86_64-secure-code.fd.xz
> 
> So about 9 MB compressed, instead of 20MB for the uncompressed sparse
> files, which is on a par with the existing ROM sizes.
> 
> We would then need a "make" rule that runs  unxz to "build" the firmware
> files. Wouldn't need to be more complicated that this:
> 
>    edk2-%.fd: edk2-%.fd.xz
>          unzx -c $< > $@
> 
>    CLEANFILES += $(wildcard edk2*.fd)

The problem with this idea is that such *.xz files are not directly
consumeable by Igor's ACPI regression test cases in qtest. The above
approach is suitable for "make install" only, not for "make check", AIUI.


I had thought of something similar earlier however. We could use
compressed qcow2 disk images for all of the *fd files here (that is,
under pc-bios/), because Igor's test code *could* consume those directly.

What blocked me there was that I didn't know if / how we could
decompress such qcow2 images as a part of "make install" (that is, from
pc-bios to the final location in the filesystem). Invoking "qemu-img
convert" (for conversion to "raw") *seems* simple enough, but what
qemu-img binary should I use? Is it guaranteed that there will be a
qemu-img binary at all, at "make install" time?

Thanks
Laszlo

Laszlo Ersek March 12, 2019, 3:58 p.m. UTC | #16

On 03/11/19 14:04, Michael S. Tsirkin wrote:
> On Mon, Mar 11, 2019 at 01:00:00PM +0000, Daniel P. Berrangé wrote:
>> On Mon, Mar 11, 2019 at 08:57:04AM -0400, Michael S. Tsirkin wrote:
>>> On Mon, Mar 11, 2019 at 10:28:01AM +0000, Daniel P. Berrangé wrote:
>>>> On Sat, Mar 09, 2019 at 02:20:06AM +0100, Philippe Mathieu-Daudé wrote:
>>>>> On 3/9/19 1:48 AM, Laszlo Ersek wrote:
>>>>>> Repo:   https://github.com/lersek/qemu.git
>>>>>> Branch: edk2_build
>>>>>>
>>>>>> This series advances the roms/edk2 submodule to the "edk2-stable201903"
>>>>>> release, and builds and captures platform firmware binaries from that
>>>>>> release. At this point they are meant to be used by both end-users and
>>>>>> by Igor's ACPI unit tests in qtest ("make check").
>>>>>>
>>>>>> Previous discussion:
>>>>>>
>>>>>>   [Qemu-devel] bundling edk2 platform firmware images with QEMU
>>>>>>   http://mid.mail-archive.com/80f0bae3-e79a-bb68-04c4-1c9c684d95b8@redhat.com
>>>>>>   https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg02601.html
>>>>>>
>>>>>> Note that the series was formatted with "--no-binary" (affecting patch
>>>>>> #8), therefore it cannot be applied with "git-am". See the remote
>>>>>> repo/branch reference near the top instead.
>>>>>>
>>>>>> Thanks,
>>>>>> Laszlo
>>>>>>
>>>>>> Laszlo Ersek (10):
>>>>>>   roms: lift "edk2-funcs.sh" from "tests/uefi-test-tools/build.sh"
>>>>>>   roms/edk2-funcs.sh: require gcc-4.8+ for building i386 and x86_64
>>>>>>   tests/uefi-test-tools/build.sh: work around TianoCore#1607
>>>>>>   roms/edk2: advance to tag edk2-stable201903
>>>>>>   roms/edk2-funcs.sh: add the qemu_edk2_get_thread_count() function
>>>>>>   roms/Makefile: replace the $(EFIROM) target with "edk2-basetools"
>>>>>>   roms: build edk2 firmware binaries and variable store templates
>>>>>>   pc-bios: add edk2 firmware binaries and variable store templates
>>>>>>   pc-bios: document the edk2 firmware images; add firmware descriptors
>>>>>>   Makefile: install the edk2 firmware images and their descriptors
>>>>>>
>>>>>>  Makefile                                       |  17 +-
>>>>>>  pc-bios/README                                 |  11 +
>>>>>>  pc-bios/descriptors/50-edk2-i386-secure.json   |  34 +++
>>>>>>  pc-bios/descriptors/50-edk2-x86_64-secure.json |  35 +++
>>>>>>  pc-bios/descriptors/60-edk2-aarch64.json       |  31 +++
>>>>>>  pc-bios/descriptors/60-edk2-arm.json           |  31 +++
>>>>>>  pc-bios/descriptors/60-edk2-i386.json          |  33 +++
>>>>>>  pc-bios/descriptors/60-edk2-x86_64.json        |  34 +++
>>>>>>  pc-bios/edk2-aarch64-code.fd                   | Bin 0 -> 67108864 bytes
>>>>>>  pc-bios/edk2-arm-code.fd                       | Bin 0 -> 67108864 bytes
>>>>>>  pc-bios/edk2-arm-vars.fd                       | Bin 0 -> 67108864 bytes
>>>>>
>>>>> GitHub moans here:
>>>>>
>>>>> remote: warning: GH001: Large files detected. You may want to try Git
>>>>> Large File Storage - https://git-lfs.github.com.
>>>>> remote: warning: See http://git.io/iEPt8g for more information.
>>>>> remote: warning: File pc-bios/edk2-arm-vars.fd is 64.00 MB; this is
>>>>> larger than GitHub's recommended maximum file size of 50.00 MB
>>>>> remote: warning: File pc-bios/edk2-arm-code.fd is 64.00 MB; this is
>>>>> larger than GitHub's recommended maximum file size of 50.00 MB
>>>>> remote: warning: File pc-bios/edk2-aarch64-code.fd is 64.00 MB; this is
>>>>> larger than GitHub's recommended maximum file size of 50.00 MB
>>>>
>>>> I wonder if this is a such that github isn't handling sparse files
>>>> well, or if they just blindly do this check before they look at the
>>>> actual required storage for the files.
>>>>
>>>> Regards,
>>>> Daniel
>>>
>>>
>>> Right. But really: can we keep these around compressed?
>>
>> I think it is viable for us to xz compress the images that we store in
>> git & just let make "build"  the uncompressed images when needed.
>>
>> Regards,
>> Daniel
> 
> Right that's the simplest approach. OTOH we do link with zlib already,
> so we could support actual compressed firmware too. Not sure it's worth
> it.

Let me attempt a summary.

(1) The packed git object footprint is 9MB.

(2) Same applies to git-push/git-pull.

(3) git-checkout is sparse, *if* you use recent enough git, and a
filesystem with support for holes.

(4) If everyone prefers some kind of compression around the edk2*fd
files, then we should use qcow2 with gzip compression. *xz is not
directly consumable to qtest. With qcow2 under pc-bios however, the
challenge for me is the "make install" logic that needs to call
"qemu-img convert", so we place the raw files in the filesystem (for
consumption by libvirt, for example). Question: *what* qemu-img exactly?

Thanks
Laszlo

Daniel P. Berrangé March 12, 2019, 3:59 p.m. UTC | #17

On Tue, Mar 12, 2019 at 04:40:45PM +0100, Laszlo Ersek wrote:
> On 03/11/19 11:35, Daniel P. Berrangé wrote:
> > On Sun, Mar 10, 2019 at 12:21:55PM +0100, Philippe Mathieu-Daudé wrote:
> >> On 3/10/19 4:56 AM, Michael S. Tsirkin wrote:
> >>> On Sat, Mar 09, 2019 at 01:48:16AM +0100, Laszlo Ersek wrote:
> >>>> Repo:   https://github.com/lersek/qemu.git
> >>>> Branch: edk2_build
> >>>>
> >>>> This series advances the roms/edk2 submodule to the "edk2-stable201903"
> >>>> release, and builds and captures platform firmware binaries from that
> >>>> release. At this point they are meant to be used by both end-users and
> >>>> by Igor's ACPI unit tests in qtest ("make check").
> >>>>
> >>>> Previous discussion:
> >>>>
> >>>>   [Qemu-devel] bundling edk2 platform firmware images with QEMU
> >>>>   http://mid.mail-archive.com/80f0bae3-e79a-bb68-04c4-1c9c684d95b8@redhat.com
> >>>>   https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg02601.html
> >>
> >> There David raised a concern about "[adding] ~206 MB of binaries to the
> >> pc-bios directory". I'm also worried.
> >>
> >> GitHub kindly suggest to use git-lfs. It is an extra dependency I'd
> >> rather strongly avoid (because we support a wide range of host OS, each
> >> using a wide types of filesystems).
> >>
> >> What about storing those binaries on a file server (http/ftp) altogether
> >> with a file containing its hashed digest (SHA1/SHA256)? Then we already
> >> have all the required tools to fetch and verify those blob roms with the
> >> build system.
> >> Or we could store the hashes in the QEMU repository too.
> > 
> > A simple approach would simply be to 'xz' compress all the edk*.fd
> > files but still store them in git. They're already opaque binary
> > files, so replacing one binary format with another binary format
> > is no big deal IMHO
> > 
> > $ ls -alsh edk2-*
> > 1.1M -rw-rw-r--. 1 berrange berrange 1.1M Mar 11 10:29 edk2-aarch64-code.fd.xz
> > 2.1M -rw-rw-r--. 1 berrange berrange  64M Mar 11 10:29 edk2-arm-code.fd
> > 772K -rw-rw-r--. 1 berrange berrange  64M Mar 11 10:29 edk2-arm-vars.fd
> > 3.5M -rw-rw-r--. 1 berrange berrange 3.5M Mar 11 10:29 edk2-i386-code.fd
> > 3.5M -rw-rw-r--. 1 berrange berrange 3.5M Mar 11 10:29 edk2-i386-secure-code.fd
> > 528K -rw-rw-r--. 1 berrange berrange 528K Mar 11 10:29 edk2-i386-vars.fd
> >  12K -rw-rw-r--. 1 berrange berrange  11K Mar 11 10:29 edk2-licenses.txt
> > 3.5M -rw-rw-r--. 1 berrange berrange 3.5M Mar 11 10:29 edk2-x86_64-code.fd
> > 3.5M -rw-rw-r--. 1 berrange berrange 3.5M Mar 11 10:29 edk2-x86_64-secure-code.fd
> > 
> > This gives a 50% disk space saving over the sparse size:
> > 
> > $ ls -alsh edk2-*
> > 1.1M -rw-rw-r--. 1 berrange berrange 1.1M Mar 11 10:29 edk2-aarch64-code.fd.xz
> > 1.1M -rw-rw-r--. 1 berrange berrange 1.1M Mar 11 10:29 edk2-arm-code.fd.xz
> >  12K -rw-rw-r--. 1 berrange berrange 9.8K Mar 11 10:29 edk2-arm-vars.fd.xz
> > 1.6M -rw-rw-r--. 1 berrange berrange 1.6M Mar 11 10:29 edk2-i386-code.fd.xz
> > 1.8M -rw-rw-r--. 1 berrange berrange 1.8M Mar 11 10:29 edk2-i386-secure-code.fd.xz
> > 4.0K -rw-rw-r--. 1 berrange berrange  320 Mar 11 10:29 edk2-i386-vars.fd.xz
> >  12K -rw-rw-r--. 1 berrange berrange  11K Mar 11 10:29 edk2-licenses.txt
> > 1.6M -rw-rw-r--. 1 berrange berrange 1.6M Mar 11 10:29 edk2-x86_64-code.fd.xz
> > 1.9M -rw-rw-r--. 1 berrange berrange 1.9M Mar 11 10:29 edk2-x86_64-secure-code.fd.xz
> > 
> > So about 9 MB compressed, instead of 20MB for the uncompressed sparse
> > files, which is on a par with the existing ROM sizes.
> > 
> > We would then need a "make" rule that runs  unxz to "build" the firmware
> > files. Wouldn't need to be more complicated that this:
> > 
> >    edk2-%.fd: edk2-%.fd.xz
> >          unzx -c $< > $@
> > 
> >    CLEANFILES += $(wildcard edk2*.fd)
> 
> The problem with this idea is that such *.xz files are not directly
> consumeable by Igor's ACPI regression test cases in qtest. The above
> approach is suitable for "make install" only, not for "make check", AIUI.

I was suggesting that this rule to uncompress is wired into the
default "make" target actually. Probably by just marking them
as generated files, so they are always built. This will make
it work for tests, as well as developers running from the source
tree.


Regards,
Daniel

Laszlo Ersek March 12, 2019, 7:30 p.m. UTC | #18

On 03/12/19 16:59, Daniel P. Berrangé wrote:
> On Tue, Mar 12, 2019 at 04:40:45PM +0100, Laszlo Ersek wrote:
>> On 03/11/19 11:35, Daniel P. Berrangé wrote:
>>> On Sun, Mar 10, 2019 at 12:21:55PM +0100, Philippe Mathieu-Daudé wrote:
>>>> On 3/10/19 4:56 AM, Michael S. Tsirkin wrote:
>>>>> On Sat, Mar 09, 2019 at 01:48:16AM +0100, Laszlo Ersek wrote:
>>>>>> Repo:   https://github.com/lersek/qemu.git
>>>>>> Branch: edk2_build
>>>>>>
>>>>>> This series advances the roms/edk2 submodule to the "edk2-stable201903"
>>>>>> release, and builds and captures platform firmware binaries from that
>>>>>> release. At this point they are meant to be used by both end-users and
>>>>>> by Igor's ACPI unit tests in qtest ("make check").
>>>>>>
>>>>>> Previous discussion:
>>>>>>
>>>>>>   [Qemu-devel] bundling edk2 platform firmware images with QEMU
>>>>>>   http://mid.mail-archive.com/80f0bae3-e79a-bb68-04c4-1c9c684d95b8@redhat.com
>>>>>>   https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg02601.html
>>>>
>>>> There David raised a concern about "[adding] ~206 MB of binaries to the
>>>> pc-bios directory". I'm also worried.
>>>>
>>>> GitHub kindly suggest to use git-lfs. It is an extra dependency I'd
>>>> rather strongly avoid (because we support a wide range of host OS, each
>>>> using a wide types of filesystems).
>>>>
>>>> What about storing those binaries on a file server (http/ftp) altogether
>>>> with a file containing its hashed digest (SHA1/SHA256)? Then we already
>>>> have all the required tools to fetch and verify those blob roms with the
>>>> build system.
>>>> Or we could store the hashes in the QEMU repository too.
>>>
>>> A simple approach would simply be to 'xz' compress all the edk*.fd
>>> files but still store them in git. They're already opaque binary
>>> files, so replacing one binary format with another binary format
>>> is no big deal IMHO
>>>
>>> $ ls -alsh edk2-*
>>> 1.1M -rw-rw-r--. 1 berrange berrange 1.1M Mar 11 10:29 edk2-aarch64-code.fd.xz
>>> 2.1M -rw-rw-r--. 1 berrange berrange  64M Mar 11 10:29 edk2-arm-code.fd
>>> 772K -rw-rw-r--. 1 berrange berrange  64M Mar 11 10:29 edk2-arm-vars.fd
>>> 3.5M -rw-rw-r--. 1 berrange berrange 3.5M Mar 11 10:29 edk2-i386-code.fd
>>> 3.5M -rw-rw-r--. 1 berrange berrange 3.5M Mar 11 10:29 edk2-i386-secure-code.fd
>>> 528K -rw-rw-r--. 1 berrange berrange 528K Mar 11 10:29 edk2-i386-vars.fd
>>>  12K -rw-rw-r--. 1 berrange berrange  11K Mar 11 10:29 edk2-licenses.txt
>>> 3.5M -rw-rw-r--. 1 berrange berrange 3.5M Mar 11 10:29 edk2-x86_64-code.fd
>>> 3.5M -rw-rw-r--. 1 berrange berrange 3.5M Mar 11 10:29 edk2-x86_64-secure-code.fd
>>>
>>> This gives a 50% disk space saving over the sparse size:
>>>
>>> $ ls -alsh edk2-*
>>> 1.1M -rw-rw-r--. 1 berrange berrange 1.1M Mar 11 10:29 edk2-aarch64-code.fd.xz
>>> 1.1M -rw-rw-r--. 1 berrange berrange 1.1M Mar 11 10:29 edk2-arm-code.fd.xz
>>>  12K -rw-rw-r--. 1 berrange berrange 9.8K Mar 11 10:29 edk2-arm-vars.fd.xz
>>> 1.6M -rw-rw-r--. 1 berrange berrange 1.6M Mar 11 10:29 edk2-i386-code.fd.xz
>>> 1.8M -rw-rw-r--. 1 berrange berrange 1.8M Mar 11 10:29 edk2-i386-secure-code.fd.xz
>>> 4.0K -rw-rw-r--. 1 berrange berrange  320 Mar 11 10:29 edk2-i386-vars.fd.xz
>>>  12K -rw-rw-r--. 1 berrange berrange  11K Mar 11 10:29 edk2-licenses.txt
>>> 1.6M -rw-rw-r--. 1 berrange berrange 1.6M Mar 11 10:29 edk2-x86_64-code.fd.xz
>>> 1.9M -rw-rw-r--. 1 berrange berrange 1.9M Mar 11 10:29 edk2-x86_64-secure-code.fd.xz
>>>
>>> So about 9 MB compressed, instead of 20MB for the uncompressed sparse
>>> files, which is on a par with the existing ROM sizes.
>>>
>>> We would then need a "make" rule that runs  unxz to "build" the firmware
>>> files. Wouldn't need to be more complicated that this:
>>>
>>>    edk2-%.fd: edk2-%.fd.xz
>>>          unzx -c $< > $@
>>>
>>>    CLEANFILES += $(wildcard edk2*.fd)
>>
>> The problem with this idea is that such *.xz files are not directly
>> consumeable by Igor's ACPI regression test cases in qtest. The above
>> approach is suitable for "make install" only, not for "make check", AIUI.
> 
> I was suggesting that this rule to uncompress is wired into the
> default "make" target actually.

Ah, I see.

We might have to mark these "side artifacts" as .SECONDARY then, lest
make remove them in the end.

https://www.gnu.org/software/make/manual/make.html#Chained-Rules

Not entirely sure yet, I'd have to see in practice.

> Probably by just marking them
> as generated files, so they are always built. This will make
> it work for tests, as well as developers running from the source
> tree.

I'm quite lost in the top level Makefile -- this idea is more complex
than just extending INSTALL_BLOBS / BLOBS.

What target should I make dependent on the uncompressed fd files so that
"make" process them for both "make" (which is "make all" I guess) and
"make check"?

Thanks
Laszlo

Philippe Mathieu-Daudé March 12, 2019, 11:09 p.m. UTC | #19

On 3/12/19 4:33 PM, Laszlo Ersek wrote:
> On 03/10/19 12:21, Philippe Mathieu-Daudé wrote:
>> On 3/10/19 4:56 AM, Michael S. Tsirkin wrote:
>>> On Sat, Mar 09, 2019 at 01:48:16AM +0100, Laszlo Ersek wrote:
>>>> Repo:   https://github.com/lersek/qemu.git
>>>> Branch: edk2_build
>>>>
>>>> This series advances the roms/edk2 submodule to the "edk2-stable201903"
>>>> release, and builds and captures platform firmware binaries from that
>>>> release. At this point they are meant to be used by both end-users and
>>>> by Igor's ACPI unit tests in qtest ("make check").
>>>>
>>>> Previous discussion:
>>>>
>>>>   [Qemu-devel] bundling edk2 platform firmware images with QEMU
>>>>   http://mid.mail-archive.com/80f0bae3-e79a-bb68-04c4-1c9c684d95b8@redhat.com
>>>>   https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg02601.html
>>
>> There David raised a concern about "[adding] ~206 MB of binaries to the
>> pc-bios directory".
> 
> (I think the concern was voiced by Dan first.)
> 
>> I'm also worried.
> 
> I'm not. The actual data size, both to transfer (push/pull) and to store
> in the .git subdirectory, is ~9MB. (As I explained earlier.)
> 
> git-checkout *could* in theory eat more space, but, in my testing, it
> doesn't; for me, git-checkout punches holes in the checked-out files.
> Please see the earlier discussion for that reference too. (Note: I'm on
> ext4, so nothing enterprisey like xfs.)
> 
>> GitHub kindly suggest to use git-lfs.
> 
> Just ignore that -- not because it's another dependency, but because
> external storage for these blobs is actually not required.
> 
>> It is an extra dependency I'd
>> rather strongly avoid (because we support a wide range of host OS, each
>> using a wide types of filesystems).
>>
>> What about storing those binaries on a file server (http/ftp) altogether
>> with a file containing its hashed digest (SHA1/SHA256)? Then we already
>> have all the required tools to fetch and verify those blob roms with the
>> build system.
>> Or we could store the hashes in the QEMU repository too.
> 
> Let's not implement "fedpkg sources" within upstream QEMU, unless we
> really must.
> 
>>
>>>> Note that the series was formatted with "--no-binary" (affecting patch
>>>> #8), therefore it cannot be applied with "git-am". See the remote
>>>> repo/branch reference near the top instead.
>>>>
>>>> Thanks,
>>>> Laszlo
>>>
>>> High time IMO.
>>
>> :)
>>
>>> Reviewed-by: Michael S. Tsirkin <mst@redhat.com>
>>>
>>> Laszlo I suggest you add an entry to MAINTAINERS
>>> and start doing pull requests.
> 
> I'm glad to do that (in the hope that updates will be mostly painless,
> regarding the build scaffolding).
> 
>>
>> This is the entry I added here:
>> https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg02967.html
> 
> Hmm. That hunk is insufficient (it doesn't cover the files being added
> here), but if I add another hunk myself, then we'll have redundancy.
> 
> In other words, this is a conflict. How should we resolve it? Should I
> add a separate patch at the end, and let Michael drop it or resolve the
> conflict, dependent on the order in which the series end up being merged?

Likely yours first.

These two lines should match your series:

F: pc-bios/descriptors/*edk2*
F: pc-bios/*edk2*

>>
>>>
>>> Peter, what do you say? OK with you?
>>
>> Since this series doesn't change the QEMU binaries built, it looks OK to
>> me to merge it past soft freeze (as we do we tests/CI), this way it get
>> merged with the final EDK2 release tag.
>> Else we can merge it next week, and update the EDK2 submodule tag
>> previous QEMU release.
> 
> I don't understand this paragraph -- the edk2 submodule commit reference
> is already the final one I have targeted for this work, namely
> "edk2-stable201903". (I had worried edk2 would suffer a delay in
> publishing the tag, but ultimately the community managed it just in time.)

"If edk2-stable201903 tag were delayed, this series is still OK to get
merged for the hard freeze (next tuesday)."

> Thanks
> Laszlo

Daniel P. Berrangé March 13, 2019, 10:11 a.m. UTC | #20

On Tue, Mar 12, 2019 at 08:30:02PM +0100, Laszlo Ersek wrote:
> On 03/12/19 16:59, Daniel P. Berrangé wrote:
> > On Tue, Mar 12, 2019 at 04:40:45PM +0100, Laszlo Ersek wrote:
> >> On 03/11/19 11:35, Daniel P. Berrangé wrote:
> >>> On Sun, Mar 10, 2019 at 12:21:55PM +0100, Philippe Mathieu-Daudé wrote:
> >>>> On 3/10/19 4:56 AM, Michael S. Tsirkin wrote:
> >>>>> On Sat, Mar 09, 2019 at 01:48:16AM +0100, Laszlo Ersek wrote:
> >>>>>> Repo:   https://github.com/lersek/qemu.git
> >>>>>> Branch: edk2_build
> >>>>>>
> >>>>>> This series advances the roms/edk2 submodule to the "edk2-stable201903"
> >>>>>> release, and builds and captures platform firmware binaries from that
> >>>>>> release. At this point they are meant to be used by both end-users and
> >>>>>> by Igor's ACPI unit tests in qtest ("make check").
> >>>>>>
> >>>>>> Previous discussion:
> >>>>>>
> >>>>>>   [Qemu-devel] bundling edk2 platform firmware images with QEMU
> >>>>>>   http://mid.mail-archive.com/80f0bae3-e79a-bb68-04c4-1c9c684d95b8@redhat.com
> >>>>>>   https://lists.gnu.org/archive/html/qemu-devel/2019-03/msg02601.html
> >>>>
> >>>> There David raised a concern about "[adding] ~206 MB of binaries to the
> >>>> pc-bios directory". I'm also worried.
> >>>>
> >>>> GitHub kindly suggest to use git-lfs. It is an extra dependency I'd
> >>>> rather strongly avoid (because we support a wide range of host OS, each
> >>>> using a wide types of filesystems).
> >>>>
> >>>> What about storing those binaries on a file server (http/ftp) altogether
> >>>> with a file containing its hashed digest (SHA1/SHA256)? Then we already
> >>>> have all the required tools to fetch and verify those blob roms with the
> >>>> build system.
> >>>> Or we could store the hashes in the QEMU repository too.
> >>>
> >>> A simple approach would simply be to 'xz' compress all the edk*.fd
> >>> files but still store them in git. They're already opaque binary
> >>> files, so replacing one binary format with another binary format
> >>> is no big deal IMHO
> >>>
> >>> $ ls -alsh edk2-*
> >>> 1.1M -rw-rw-r--. 1 berrange berrange 1.1M Mar 11 10:29 edk2-aarch64-code.fd.xz
> >>> 2.1M -rw-rw-r--. 1 berrange berrange  64M Mar 11 10:29 edk2-arm-code.fd
> >>> 772K -rw-rw-r--. 1 berrange berrange  64M Mar 11 10:29 edk2-arm-vars.fd
> >>> 3.5M -rw-rw-r--. 1 berrange berrange 3.5M Mar 11 10:29 edk2-i386-code.fd
> >>> 3.5M -rw-rw-r--. 1 berrange berrange 3.5M Mar 11 10:29 edk2-i386-secure-code.fd
> >>> 528K -rw-rw-r--. 1 berrange berrange 528K Mar 11 10:29 edk2-i386-vars.fd
> >>>  12K -rw-rw-r--. 1 berrange berrange  11K Mar 11 10:29 edk2-licenses.txt
> >>> 3.5M -rw-rw-r--. 1 berrange berrange 3.5M Mar 11 10:29 edk2-x86_64-code.fd
> >>> 3.5M -rw-rw-r--. 1 berrange berrange 3.5M Mar 11 10:29 edk2-x86_64-secure-code.fd
> >>>
> >>> This gives a 50% disk space saving over the sparse size:
> >>>
> >>> $ ls -alsh edk2-*
> >>> 1.1M -rw-rw-r--. 1 berrange berrange 1.1M Mar 11 10:29 edk2-aarch64-code.fd.xz
> >>> 1.1M -rw-rw-r--. 1 berrange berrange 1.1M Mar 11 10:29 edk2-arm-code.fd.xz
> >>>  12K -rw-rw-r--. 1 berrange berrange 9.8K Mar 11 10:29 edk2-arm-vars.fd.xz
> >>> 1.6M -rw-rw-r--. 1 berrange berrange 1.6M Mar 11 10:29 edk2-i386-code.fd.xz
> >>> 1.8M -rw-rw-r--. 1 berrange berrange 1.8M Mar 11 10:29 edk2-i386-secure-code.fd.xz
> >>> 4.0K -rw-rw-r--. 1 berrange berrange  320 Mar 11 10:29 edk2-i386-vars.fd.xz
> >>>  12K -rw-rw-r--. 1 berrange berrange  11K Mar 11 10:29 edk2-licenses.txt
> >>> 1.6M -rw-rw-r--. 1 berrange berrange 1.6M Mar 11 10:29 edk2-x86_64-code.fd.xz
> >>> 1.9M -rw-rw-r--. 1 berrange berrange 1.9M Mar 11 10:29 edk2-x86_64-secure-code.fd.xz
> >>>
> >>> So about 9 MB compressed, instead of 20MB for the uncompressed sparse
> >>> files, which is on a par with the existing ROM sizes.
> >>>
> >>> We would then need a "make" rule that runs  unxz to "build" the firmware
> >>> files. Wouldn't need to be more complicated that this:
> >>>
> >>>    edk2-%.fd: edk2-%.fd.xz
> >>>          unzx -c $< > $@
> >>>
> >>>    CLEANFILES += $(wildcard edk2*.fd)
> >>
> >> The problem with this idea is that such *.xz files are not directly
> >> consumeable by Igor's ACPI regression test cases in qtest. The above
> >> approach is suitable for "make install" only, not for "make check", AIUI.
> > 
> > I was suggesting that this rule to uncompress is wired into the
> > default "make" target actually.
> 
> Ah, I see.
> 
> We might have to mark these "side artifacts" as .SECONDARY then, lest
> make remove them in the end.
> 
> https://www.gnu.org/software/make/manual/make.html#Chained-Rules
> 
> Not entirely sure yet, I'd have to see in practice.
> 
> > Probably by just marking them
> > as generated files, so they are always built. This will make
> > it work for tests, as well as developers running from the source
> > tree.
> 
> I'm quite lost in the top level Makefile -- this idea is more complex
> than just extending INSTALL_BLOBS / BLOBS.
> 
> What target should I make dependent on the uncompressed fd files so that
> "make" process them for both "make" (which is "make all" I guess) and
> "make check"?

I think it is sufficient to edit the main Makefile and add the list
of UEFI firmwares to the GENERATED_FILES  variable. Anything listed
in $GENERATED_FILES is generated before any of the other make rules
run.

Regards,
Daniel

[00/10] bundle edk2 platform firmware with QEMU

Message

Comments