docs: describe the QEMU build system structure / design

Developers who are new to QEMU, or have a background familiarity
with GNU autotools can have trouble getting their head around the
home-grown QEMU build system. This document attempts to explain
the structure / design of the configure script and the various
Makefile pieces that live across the source tree.

Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
---
 docs/build-system.txt | 493 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 493 insertions(+)
 create mode 100644 docs/build-system.txt

Reviewed from an en_US perspective, though I left alone things that are
clearly regional (e.g. 'behaviour' vs 'behavior')

On 09/22/2015 12:35 PM, Daniel P. Berrange wrote:
> Developers who are new to QEMU, or have a background familiarity
> with GNU autotools can have trouble getting their head around the
> home-grown QEMU build system. This document attempts to explain
> the structure / design of the configure script and the various
> Makefile pieces that live across the source tree.
> 
> Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
> ---
>  docs/build-system.txt | 493 ++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 493 insertions(+)
>  create mode 100644 docs/build-system.txt
> 
> diff --git a/docs/build-system.txt b/docs/build-system.txt
> new file mode 100644
> index 0000000..2209261
> --- /dev/null
> +++ b/docs/build-system.txt
> @@ -0,0 +1,493 @@
> +    The QEMU build system architecture
> +    ==================================
> +
> +This document aims to help developers understand the architecture of the
> +QEMU build system. As with projects using GNU autotools, the QEMU build
> +system has two stages, first the developer runs the "configure" script
> +to determine the local build environment characteristics, then they run
> +"make" to build the project. There is about where the similarities with
> +GNU autotools end, so try to forget what you know about them.
> +
> +
> +Stage 1: configure
> +==================
> +
> +The QEMU configure script is written directly in shell, and should be
> +compatible with any POSIX shell, hence it uses #!/bin/sh. An important
> +implication of this is that it is important to avoid using bash-isms on
> +development platforms where bash is the primary host.
> +
> +In contrast to autoconf scripts, QEMU's configure is expected to be
> +silent while it is checking for features. It will only display output
> +when an error occurrs, or to show the final feature enablement summary

occurs.

> +on completion.
> +
> +Adding new checks to the configure script usually comprises the
> +following tasks
> +

colon for consistency with other lists below

> + - Initialize one or more variables with the default feature state.
> +
> +   Ideally features should auto-detect whether they are present,
> +   so try to avoid hardcoding the initial state to either enabled
> +   or disabled, as that forces the user to pass a --{dis,en}able-XXX
> +   flag on every invokation of configure
> +

invocation?

> + - Add support to the command line arg parser to handle any new
> +   --{dis,en}able-XXX flags required by the feature XXX
> +
> + - Add information to the help output message to report on the new
> +   feature flag.
> +

Remove period, or add to the other list items for consistency. My
personal preference is to use the period for any sentences with proper
grammatical structure, omitting it for simple list items.

> + - Add code to perform the actual feature check. As noted above, try to
> +   be fully dynamic in checking enablement/disablement
> +
> + - Add code to print out the feature status in the configure summary
> +   upon completion
> +
> + - Add any new makefile variables to $config_host_mak on completion
> +
> +
> +Taking (a simplified version of) the probe for gnutls from configure,
> +we have the following pieces:
> +
> +  # Initial variable state
> +  gnutls=""
> +
> +  ..snip..
> +
> +
> +  # Configure flag processing
> +  --disable-gnutls) gnutls="no"
> +  ;;
> +  --enable-gnutls) gnutls="yes"
> +  ;;
> +
> +  ..snip..
> +
> +
> +  # Help output feature message
> +  gnutls          GNUTLS cryptography support
> +
> +  ..snip..
> +
> +
> +  # Test for gnutls
> +  if test "$gnutls" != "no"; then
> +     if ! $pkg_config --exists "gnutls"; then
> +        gnutls_cflags=`$pkg_config --cflags gnutls`
> +        gnutls_libs=`$pkg_config --libs gnutls`
> +        libs_softmmu="$gnutls_libs $libs_softmmu"
> +        libs_tools="$gnutls_libs $libs_tools"
> +        QEMU_CFLAGS="$QEMU_CFLAGS $gnutls_cflags"
> +        gnutls="yes"
> +     elif test "$gnutls" = "yes"; then
> +        feature_not_found "gnutls" "Install gnutls devel"
> +     else
> +        gnutls="no"
> +     fi
> +  fi
> +
> +  ..snip..
> +
> +
> +  # Completion feature summary
> +  echo "GNUTLS support    $gnutls"
> +
> +  ..snip..
> +
> +
> +  # Define make variables
> +  if test "$gnutls" = "yes" ; then
> +     echo "CONFIG_GNUTLS=y" >> $config_host_mak
> +  fi
> +
> +
> +Helper functions
> +----------------
> +
> +The configure script provides a variety of helper functions to assist
> +developers in checking for system features:
> +
> + - do_cc $ARGS...
> +
> +   Attempt to run the system C compiler passing it $ARGS...
> +
> + - do_cxx $ARGS...
> +
> +   Attempt to run the system C++ compiler passing it $ARGS...
> +
> + - compile_object $CFLAGS
> +
> +   Attempt to compile a test program with the system C compiler using
> +   $CFLAGS. The test program must have been previously written to a file
> +   called $TMPC.
> +
> + - compile_prog $CFLAGS $LDFLAGS
> +
> +   Attempt to compile a test program with the system C compiler using
> +   $CFLAGS and link it with the system linker using $LDFLAGS. The test
> +   program must have been previously written to a file called $TMPC.
> +
> + - has $COMMAND
> +
> +   Determine if $COMMAND exists in the current environment, either as a
> +   shell builtin, or executable binary, returning 0 on success.
> +
> + - path_of $COMMAND
> +
> +   Return the fully qualified path of $COMMAND, printing it to stdout,
> +   and returning 0 on success.
> +
> + - check_define $NAME
> +
> +   Determine if the macro $NAME is defined by the system C compiler
> +
> + - check_include $NAME
> +
> +   Determine if the include $NAME file is available to the system C
> +   compiler
> +
> + - write_c_skeleton
> +
> +   Written a minimal C program main() function to the temporary file
> +   indicated by $TMPC
> +
> + - feature_not_found $NAME $REMEDY
> +
> +   Print a message to stderr that the feature $NAME was not available
> +   on the system, suggesting the user try $REMEDY to address the
> +   problem.
> +
> + - error_exit $MESSAGE $MORE...
> +
> +   Print $MESSAGE to stderr, followed by $MORE... and then exit from the
> +   configure script with non-zero status
> +
> + - query_pkg_config $ARGS...
> +
> +   Run pkg-config passing it $ARGS. If QEMU is doing a static build,
> +   then --static will be automatically added to $ARGS
> +
> +
> +Stage 2: makefiles
> +==================
> +
> +Although the source code is spread across multiple subdirectories, the
> +build system should be considered largely non-recursive in nature, in
> +contrast to common practices seen with automake. There is some recursive
> +invokation of make, but this is related to the things being built,

invocation, again.

> +rather than the source directory structure.
> +
> +QEMU currently supports both VPATH and non-VPATH builds, so there are
> +three general ways to invoke configure & perform a build.
> +
> + - VPATH, build artifacts outside of QEMU source tree entirely
> +
> +     cd ../
> +     mkdir build
> +     cd build
> +     ../qemu/configure
> +     make
> +
> + - VPATH, build artifacts in a subdir of QEMU source tree
> +
> +     mkdir build
> +     cd build
> +     ../configure
> +     make
> +
> + - non-VPATH, build artifacts everywhere
> +
> +     ./configure
> +     make
> +
> +The QEMU maintainers generally recommend that a VPATH build is used by
> +developers. Patches to QEMU are expected to ensure VPATH build still
> +works.
> +
> +
> +Module structure
> +----------------
> +
> +There are a number of key outputs of the QEMU build system
> +

Colon again.

> + - Tools - qemu-img, qemu-nbd, qga (guest agent), etc
> + - System emulators - qemu-system-$ARCH
> + - Userspace emulators - qemu-$ARCH
> + - Unit tests
> +
> +The source code is highly modularized, split across many files to
> +facilitate building of all of these components with as little duplicated
> +compilation as possible. There can be considered to be two distinct
> +groups of files, those which are independant of the QEMU emulation
> +target and those which are dependant on the QEMU emulation target.
> +

independent, dependent: replaced for all below occurrences as well.

> +In the target independant set lives various general purpose helper code,

I might hyphenate this as 'target-independent.'

> +such as error handling infrastructure, standard data structures,
> +platform portability wrapper functions, etc. This code can be compiled
> +once only and the .o files linked into all output binaries.
> +
> +In the target dependant set lives CPU emulation, device emulation and
> +much glue code. This code is generally compiled multiple times, once for
> +each target architecture being built.
> +
> +The target independant code that is used by all binaries is built into a
> +static archive called libqemuutil.a, which is then linked to all the
> +binaries. Due to ongoing incomplete refactoring, some of the code in
> +libqemuutil.a depends on other functions that are not available in all
> +QEMU binaries. To deal with this there is a second library called
> +libqemustub.a which provide dummy stubs for all these functions. These
> +will get lazy linked into the binary if the real implementation is not
> +present. Thus any time a binary links to libqemuutil.a, it should also
> +be made to link libqemustub.a. eg
> +
> + qemu-img$(EXESUF): qemu-img.o ..snip.. libqemuutil.a libqemustub.a
> +
> +
> +Windows platform portability
> +----------------------------
> +
> +On Windows all binaries have a .exe suffix, so all the Makefile rules

I guess you pronounce the 'dot' :)

> +which create binaries must include the $(EXESUF) variable on the binary
> +name. eg

'e.g.' here and everywhere subsequent.

> +
> + qemu-img$(EXESUF): qemu-img.o ..snip..
> +
> +This expands to '.exe' on Windows, or '' on other platforms.
> +
> +A further complication for the system and userspace emulator binaries is
> +that two separate binaries need to be generated.
> +
> +The main binary (eg qemu-system-x86_64.exe) is linked against the
> +Windows console runtime subsystem. These are expected to be run from a
> +command prompt window, and so will print stderr to the console that
> +launched them.
> +
> +The second binary generated has a 'w' on the end of its name (eg
> +qemu-system-x86_64w.exe) and is linked against the Windows graphical
> +runtime subsystem. These are expected to be run directly from the
> +desktop and will open up a dedicated console window for stderr output.
> +
> +The Makefile.target will generate the binary for the graphical subsystem
> +first, and then use objcopy to relink it against the console subsystem
> +to generate the second binary.
> +
> +
> +Object variable naming
> +----------------------
> +
> +The QEMU convention is to define variables to list different groups of
> +object files. These are named with the convention $PREFIX-y. For example
> +the libqemuutil.a file will be linked with all objects listed in a
> +variable 'util-y'. So, for example, util/Makefile.obj will contain a set
> +of definitions looking like
> +
> +  util-obj-y += bitmap.o bitops.o hbitmap.o
> +  util-obj-y += fifo8.o
> +  util-obj-y += acl.o
> +  util-obj-y += error.o qemu-error.o
> +
> +When there is an object file which needs to be conditionally built based
> +on some characteristic of the host system, the configure script will
> +define a variable for the conditional. For example, on Windows it will
> +define $(CONFIG_POSIX) with a value of 'n' and $(CONFIG_WIN32) with a
> +value of 'y'. It is now possible to use the config variables when
> +listing object files. For example,
> +
> +  util-obj-$(CONFIG_WIN32) += oslib-win32.o qemu-thread-win32.o
> +  util-obj-$(CONFIG_POSIX) += oslib-posix.o qemu-thread-posix.o
> +
> +On Windows this expands to
> +
> +  util-obj-y += oslib-win32.o qemu-thread-win32.o
> +  util-obj-n += oslib-posix.o qemu-thread-posix.o
> +
> +Since libqemutil.a links in $(util-obj-y), the POSIX specific files
> +listed against $(util-obj-n) are ignored on the Windows platform builds.
> +
> +
> +CFLAGS / LDFLAGS / LIBS handling
> +--------------------------------
> +
> +There are many different binaries being built with differing purposes,
> +and some of them might even be 3rd party libraries pulled in via git
> +submodules. As such the use of the global CFLAGS / LDFLAGS variables is
> +generally avoided in QEMU, since they would apply to too many build
> +targets.
> +
> +Flags that are needed by any QEMU code (ie everything *except* GIT

i.e.

> +submodule projects) are put in $(QEMU_CFLAGS) variable. There are no
> +corresponding $(QEMU_LIBS)/$(QEMU_LDFLAGS) variables, instead there are
> +a couple of more targetted variables. $(libs_softmmu) is used for

targeted

> +libraries that must be linked to system emulator targets, $(libs_tools)
> +is used for tools like qemu-img, qemu-nbd, etc and $(libs_qga) is used
> +for the QEMU guest agent. There is currently no variable for the
> +userspace emulator targets.
> +
> +In addition to these variables, it is possible to provide cflags and
> +libs against individual source code files, by defining variables of the
> +form $FILENAME-cflags and $FILENAME-libs. For example, the curl block
> +driver needs to link to the libcurl library, so block/Makefile defines
> +some variables:
> +
> +  curl.o-cflags      := $(CURL_CFLAGS)
> +  curl.o-libs        := $(CURL_LIBS)
> +
> +The scope is a little different between the two variables. The libs get
> +used when linking any target binary that includes the curl.o object
> +file, while the cflags get used when compiling the curl.c file only.
> +
> +
> +Statically defined files
> +------------------------
> +
> +The following key files are statically defined in the source tree, with
> +the rules needed to build QEMU. Their behaviour is influenced by a
> +number of dynamically created files listed later.
> +
> +- Makefile
> +
> +The main entry point used when invoking make to build all the components
> +of QEMU. The default 'all' target will naturally result in the build of
> +every component. The various tools and helper binaries are built
> +directly via a non-recursive set of rules.
> +
> +Each system/userspace emulation target needs to have a slightly
> +different set of make rules / variables. Thus, make will be recursively
> +invoked for each of the emulation targets.
> +
> +The recursive invokation will end up processing the toplevel

invocation again.

> +Makefile.target file (more on that later).
> +
> +
> +- */Makefile.objs
> +
> +Since the source code is spread across multiple directories, the rules
> +for each file are similarly modularized. Thus each subdirectory
> +containing .c files will usually also contain a Makefile.objs file.
> +These files are not directly invoked by a recursive make, but instead
> +they are imported by the top level Makefile and/or Makefile.target
> +
> +Each Makefile.objs usually just declares a set of variables listing the
> +.o files that need building from the source files in the directory. They
> +will also define any custom linker or compiler flags. For example in
> +block/Makefile.objs
> +
> +  block-obj-$(CONFIG_LIBISCSI) += iscsi.o
> +  block-obj-$(CONFIG_CURL) += curl.o
> +
> +  ..snip...
> +
> +  iscsi.o-cflags     := $(LIBISCSI_CFLAGS)
> +  iscsi.o-libs       := $(LIBISCSI_LIBS)
> +  curl.o-cflags      := $(CURL_CFLAGS)
> +  curl.o-libs        := $(CURL_LIBS)
> +
> +
> +- Makefile.target
> +
> +This file provides the entry point used to build each individual system
> +or userspace emulator target. Each enabled target has its own
> +subdirectory. For example if configure is run with the argument
> +'--target-list=x86_64-softmmu', then a sub-directory 'x86_64-softmu'
> +will be created, containing a 'Makefile' which symlinks back to
> +Makefile.target
> +
> +So when the recursive '$(MAKE) -C x86_64-softmmu' is invoked, it ends up
> +using Makefile.target for the build rules.
> +
> +
> +- rules.mak
> +
> +This file provides the generic helper rules for invoking build tools, in
> +particular the compiler and linker. This also contains the magic (hairy)
> +'unnest-vars' function which is used to merge the variable definitions
> +from all Makefile.objs in the source tree down into the main Makefile
> +context.
> +
> +
> +- default-configs/*.mak
> +
> +The files under default-configs/ control what emulated hardware is built
> +into each QEMU system and userspace emulator targets. They merely
> +contain a long list of config variable definitions. For example,
> +default-configs/x86_64-softmmu.mak has:
> +
> +  include pci.mak
> +  include sound.mak
> +  include usb.mak
> +  CONFIG_QXL=$(CONFIG_SPICE)
> +  CONFIG_VGA_ISA=y
> +  CONFIG_VGA_CIRRUS=y
> +  CONFIG_VMWARE_VGA=y
> +  CONFIG_VIRTIO_VGA=y
> +  ...snip...
> +
> +These files rarely need changing unless new devices / hardware need to
> +be enabled for a particular system/userspace emulation target
> +
> +
> +- tests/Makefile
> +
> +Rules for building the unit tests. This file is included directly by the
> +top level Makefile, so anything defined in this file will influence the
> +entire build system. Care needs to be taken when writing rules for tests
> +to ensure they only apply to the unit test execution / build.
> +
> +
> +- po/Makefile
> +
> +Rules for building and installing the binary message catalogs from the
> +text .po file sources. This almost never needs changing for any reason.
> +
> +
> +Dynamically created files
> +-------------------------
> +
> +The following files are generated dynamically by configure in order to
> +control the behaviour of the staticaly defined makefiles. This avoids

statically

> +the need for QEMU makefiles to go through any pre-processing as seen
> +with autotools, where Makefile.am generates Makefile.in which generates
> +Makefile.
> +
> +- config-host.mak
> +
> +When configure has determined the characteristics of the build host it
> +will write a long list of variables to config-host.mak file. This
> +provides the various install directories, compiler / linker flags and a
> +variable of CONFIG_* variables related to optionally enabled features.
> +This is imported by the top level Makefile in order to tailor the build
> +output.
> +
> +The variables defined here are those which are applicable to all QEMU
> +build outputs. Variables which are potentially different for each
> +emulator target are defined by the next file...
> +
> +It is also used as a dependancy checking mechanism. If make sees that

dependency

> +the modification timestamp on configure is newer than that on
> +config-host.mak, then configure will be re-run.
> +
> +
> +- $TARGET-NAME/config-target.mak
> +
> +TARGET-NAME is the name of a system or userspace emulator, for example,
> +x86_64-softmmu denotes the system emulator for the x86_64 architecture.
> +This file contains the variables which need to vary on a per-target
> +basis. For example, it will indicate whether KVM or Xen are enabled for
> +the target and any other potential custom libraries needed for linking
> +the target.
> +
> +
> +- $TARGET-NAME/config-devices.mak
> +
> +TARGET-NAME is again the name of a system or userspace emulator. The
> +config-devices.mak file is automatically generated by make using the
> +scripts/make_device_config.sh program, feeding it the
> +default-configs/$TARGET-NAME file as input.
> +
> +
> +- $TARGET-NAME/Makefile
> +
> +This is the entrypoint used when make recurses to build a single system
> +or userspace emulator target. It is merely a symlink back to the
> +Makefile.target in the top level.
> 

Thanks for writing this!

Pretending to be Eric,
--John Snow

On 09/22/15 18:35, Daniel P. Berrange wrote:
> Developers who are new to QEMU, or have a background familiarity
> with GNU autotools can have trouble getting their head around the
> home-grown QEMU build system. This document attempts to explain
> the structure / design of the configure script and the various
> Makefile pieces that live across the source tree.
> 
> Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
> ---
>  docs/build-system.txt | 493 ++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 493 insertions(+)
>  create mode 100644 docs/build-system.txt

Wow, this is awesome. Just a few comments (I'm rather a happy student of
this document than an expert on the build system):

> 
> diff --git a/docs/build-system.txt b/docs/build-system.txt
> new file mode 100644
> index 0000000..2209261
> --- /dev/null
> +++ b/docs/build-system.txt
> @@ -0,0 +1,493 @@
> +    The QEMU build system architecture
> +    ==================================
> +
> +This document aims to help developers understand the architecture of the
> +QEMU build system. As with projects using GNU autotools, the QEMU build
> +system has two stages, first the developer runs the "configure" script
> +to determine the local build environment characteristics, then they run
> +"make" to build the project. There is about where the similarities with
> +GNU autotools end, so try to forget what you know about them.
> +
> +
> +Stage 1: configure
> +==================
> +
> +The QEMU configure script is written directly in shell, and should be
> +compatible with any POSIX shell, hence it uses #!/bin/sh. An important
> +implication of this is that it is important to avoid using bash-isms on
> +development platforms where bash is the primary host.
> +
> +In contrast to autoconf scripts, QEMU's configure is expected to be
> +silent while it is checking for features. It will only display output
> +when an error occurrs, or to show the final feature enablement summary
> +on completion.
> +
> +Adding new checks to the configure script usually comprises the
> +following tasks
> +
> + - Initialize one or more variables with the default feature state.
> +
> +   Ideally features should auto-detect whether they are present,
> +   so try to avoid hardcoding the initial state to either enabled
> +   or disabled, as that forces the user to pass a --{dis,en}able-XXX
> +   flag on every invokation of configure
> +
> + - Add support to the command line arg parser to handle any new
> +   --{dis,en}able-XXX flags required by the feature XXX

Suggest to spell out --disable-XXX and --enable-XXX separately, so that
they be the first hits when someone searches this file for "--enable"
and "--disable".

> +
> + - Add information to the help output message to report on the new
> +   feature flag.
> +
> + - Add code to perform the actual feature check. As noted above, try to
> +   be fully dynamic in checking enablement/disablement
> +
> + - Add code to print out the feature status in the configure summary
> +   upon completion
> +
> + - Add any new makefile variables to $config_host_mak on completion
> +
> +
> +Taking (a simplified version of) the probe for gnutls from configure,
> +we have the following pieces:
> +
> +  # Initial variable state
> +  gnutls=""
> +
> +  ..snip..
> +
> +

suggest to add only one empty line after "..snip..".

> +  # Configure flag processing
> +  --disable-gnutls) gnutls="no"
> +  ;;
> +  --enable-gnutls) gnutls="yes"
> +  ;;
> +
> +  ..snip..
> +
> +
> +  # Help output feature message
> +  gnutls          GNUTLS cryptography support
> +
> +  ..snip..
> +
> +
> +  # Test for gnutls
> +  if test "$gnutls" != "no"; then
> +     if ! $pkg_config --exists "gnutls"; then
> +        gnutls_cflags=`$pkg_config --cflags gnutls`
> +        gnutls_libs=`$pkg_config --libs gnutls`
> +        libs_softmmu="$gnutls_libs $libs_softmmu"
> +        libs_tools="$gnutls_libs $libs_tools"
> +        QEMU_CFLAGS="$QEMU_CFLAGS $gnutls_cflags"
> +        gnutls="yes"
> +     elif test "$gnutls" = "yes"; then
> +        feature_not_found "gnutls" "Install gnutls devel"
> +     else
> +        gnutls="no"
> +     fi
> +  fi
> +
> +  ..snip..
> +
> +
> +  # Completion feature summary
> +  echo "GNUTLS support    $gnutls"
> +
> +  ..snip..
> +
> +
> +  # Define make variables
> +  if test "$gnutls" = "yes" ; then
> +     echo "CONFIG_GNUTLS=y" >> $config_host_mak
> +  fi
> +
> +
> +Helper functions
> +----------------
> +
> +The configure script provides a variety of helper functions to assist
> +developers in checking for system features:
> +
> + - do_cc $ARGS...
> +
> +   Attempt to run the system C compiler passing it $ARGS...
> +
> + - do_cxx $ARGS...
> +
> +   Attempt to run the system C++ compiler passing it $ARGS...
> +
> + - compile_object $CFLAGS
> +
> +   Attempt to compile a test program with the system C compiler using
> +   $CFLAGS. The test program must have been previously written to a file
> +   called $TMPC.
> +
> + - compile_prog $CFLAGS $LDFLAGS
> +
> +   Attempt to compile a test program with the system C compiler using
> +   $CFLAGS and link it with the system linker using $LDFLAGS. The test
> +   program must have been previously written to a file called $TMPC.
> +
> + - has $COMMAND
> +
> +   Determine if $COMMAND exists in the current environment, either as a
> +   shell builtin, or executable binary, returning 0 on success.
> +
> + - path_of $COMMAND
> +
> +   Return the fully qualified path of $COMMAND, printing it to stdout,
> +   and returning 0 on success.
> +
> + - check_define $NAME
> +
> +   Determine if the macro $NAME is defined by the system C compiler
> +
> + - check_include $NAME
> +
> +   Determine if the include $NAME file is available to the system C
> +   compiler
> +
> + - write_c_skeleton
> +
> +   Written a minimal C program main() function to the temporary file
> +   indicated by $TMPC

The other pieces of explanation use the imperative verb form; this one's
a past participle. Suggest to unify.

> +
> + - feature_not_found $NAME $REMEDY
> +
> +   Print a message to stderr that the feature $NAME was not available
> +   on the system, suggesting the user try $REMEDY to address the
> +   problem.
> +
> + - error_exit $MESSAGE $MORE...
> +
> +   Print $MESSAGE to stderr, followed by $MORE... and then exit from the
> +   configure script with non-zero status
> +
> + - query_pkg_config $ARGS...
> +
> +   Run pkg-config passing it $ARGS. If QEMU is doing a static build,
> +   then --static will be automatically added to $ARGS
> +
> +
> +Stage 2: makefiles
> +==================
> +
> +Although the source code is spread across multiple subdirectories, the
> +build system should be considered largely non-recursive in nature, in
> +contrast to common practices seen with automake. There is some recursive
> +invokation of make, but this is related to the things being built,
> +rather than the source directory structure.
> +
> +QEMU currently supports both VPATH and non-VPATH builds, so there are
> +three general ways to invoke configure & perform a build.
> +
> + - VPATH, build artifacts outside of QEMU source tree entirely
> +
> +     cd ../
> +     mkdir build
> +     cd build
> +     ../qemu/configure
> +     make
> +
> + - VPATH, build artifacts in a subdir of QEMU source tree
> +
> +     mkdir build
> +     cd build
> +     ../configure
> +     make
> +
> + - non-VPATH, build artifacts everywhere
> +
> +     ./configure
> +     make
> +
> +The QEMU maintainers generally recommend that a VPATH build is used by
> +developers. Patches to QEMU are expected to ensure VPATH build still
> +works.
> +
> +
> +Module structure
> +----------------
> +
> +There are a number of key outputs of the QEMU build system
> +
> + - Tools - qemu-img, qemu-nbd, qga (guest agent), etc
> + - System emulators - qemu-system-$ARCH
> + - Userspace emulators - qemu-$ARCH
> + - Unit tests
> +
> +The source code is highly modularized, split across many files to
> +facilitate building of all of these components with as little duplicated
> +compilation as possible. There can be considered to be two distinct
> +groups of files, those which are independant of the QEMU emulation

wiktionary suggests "independant" is a misspelling, it should be
"independent"

> +target and those which are dependant on the QEMU emulation target.

not a misspelling here, just "obsolete", according to wiktionary

> +
> +In the target independant set lives various general purpose helper code,

see above

> +such as error handling infrastructure, standard data structures,
> +platform portability wrapper functions, etc. This code can be compiled
> +once only and the .o files linked into all output binaries.
> +
> +In the target dependant set lives CPU emulation, device emulation and

ditto

> +much glue code. This code is generally compiled multiple times, once for
> +each target architecture being built.
> +
> +The target independant code that is used by all binaries is built into a

ditto

> +static archive called libqemuutil.a, which is then linked to all the
> +binaries. Due to ongoing incomplete refactoring, some of the code in
> +libqemuutil.a depends on other functions that are not available in all
> +QEMU binaries. To deal with this there is a second library called
> +libqemustub.a which provide dummy stubs for all these functions. These
> +will get lazy linked into the binary if the real implementation is not
> +present. Thus any time a binary links to libqemuutil.a, it should also
> +be made to link libqemustub.a. eg
> +
> + qemu-img$(EXESUF): qemu-img.o ..snip.. libqemuutil.a libqemustub.a
> +
> +
> +Windows platform portability
> +----------------------------
> +
> +On Windows all binaries have a .exe suffix, so all the Makefile rules
> +which create binaries must include the $(EXESUF) variable on the binary
> +name. eg
> +
> + qemu-img$(EXESUF): qemu-img.o ..snip..
> +
> +This expands to '.exe' on Windows, or '' on other platforms.
> +
> +A further complication for the system and userspace emulator binaries is
> +that two separate binaries need to be generated.
> +
> +The main binary (eg qemu-system-x86_64.exe) is linked against the
> +Windows console runtime subsystem. These are expected to be run from a
> +command prompt window, and so will print stderr to the console that
> +launched them.
> +
> +The second binary generated has a 'w' on the end of its name (eg
> +qemu-system-x86_64w.exe) and is linked against the Windows graphical
> +runtime subsystem. These are expected to be run directly from the
> +desktop and will open up a dedicated console window for stderr output.
> +
> +The Makefile.target will generate the binary for the graphical subsystem
> +first, and then use objcopy to relink it against the console subsystem
> +to generate the second binary.
> +
> +
> +Object variable naming
> +----------------------
> +
> +The QEMU convention is to define variables to list different groups of
> +object files. These are named with the convention $PREFIX-y. For example
> +the libqemuutil.a file will be linked with all objects listed in a
> +variable 'util-y'. So, for example, util/Makefile.obj will contain a set
> +of definitions looking like
> +
> +  util-obj-y += bitmap.o bitops.o hbitmap.o
> +  util-obj-y += fifo8.o
> +  util-obj-y += acl.o
> +  util-obj-y += error.o qemu-error.o
> +
> +When there is an object file which needs to be conditionally built based
> +on some characteristic of the host system, the configure script will
> +define a variable for the conditional. For example, on Windows it will
> +define $(CONFIG_POSIX) with a value of 'n' and $(CONFIG_WIN32) with a
> +value of 'y'. It is now possible to use the config variables when
> +listing object files. For example,
> +
> +  util-obj-$(CONFIG_WIN32) += oslib-win32.o qemu-thread-win32.o
> +  util-obj-$(CONFIG_POSIX) += oslib-posix.o qemu-thread-posix.o
> +
> +On Windows this expands to
> +
> +  util-obj-y += oslib-win32.o qemu-thread-win32.o
> +  util-obj-n += oslib-posix.o qemu-thread-posix.o
> +
> +Since libqemutil.a links in $(util-obj-y), the POSIX specific files
> +listed against $(util-obj-n) are ignored on the Windows platform builds.
> +
> +
> +CFLAGS / LDFLAGS / LIBS handling
> +--------------------------------
> +
> +There are many different binaries being built with differing purposes,
> +and some of them might even be 3rd party libraries pulled in via git
> +submodules. As such the use of the global CFLAGS / LDFLAGS variables is
> +generally avoided in QEMU, since they would apply to too many build
> +targets.
> +
> +Flags that are needed by any QEMU code (ie everything *except* GIT
> +submodule projects) are put in $(QEMU_CFLAGS) variable. There are no
> +corresponding $(QEMU_LIBS)/$(QEMU_LDFLAGS) variables, instead there are
> +a couple of more targetted variables. $(libs_softmmu) is used for
> +libraries that must be linked to system emulator targets, $(libs_tools)
> +is used for tools like qemu-img, qemu-nbd, etc and $(libs_qga) is used
> +for the QEMU guest agent. There is currently no variable for the
> +userspace emulator targets.
> +
> +In addition to these variables, it is possible to provide cflags and
> +libs against individual source code files, by defining variables of the
> +form $FILENAME-cflags and $FILENAME-libs. For example, the curl block
> +driver needs to link to the libcurl library, so block/Makefile defines
> +some variables:
> +
> +  curl.o-cflags      := $(CURL_CFLAGS)
> +  curl.o-libs        := $(CURL_LIBS)
> +
> +The scope is a little different between the two variables. The libs get
> +used when linking any target binary that includes the curl.o object
> +file, while the cflags get used when compiling the curl.c file only.
> +
> +
> +Statically defined files
> +------------------------
> +
> +The following key files are statically defined in the source tree, with
> +the rules needed to build QEMU. Their behaviour is influenced by a
> +number of dynamically created files listed later.
> +
> +- Makefile
> +
> +The main entry point used when invoking make to build all the components
> +of QEMU. The default 'all' target will naturally result in the build of
> +every component. The various tools and helper binaries are built
> +directly via a non-recursive set of rules.
> +
> +Each system/userspace emulation target needs to have a slightly
> +different set of make rules / variables. Thus, make will be recursively
> +invoked for each of the emulation targets.
> +
> +The recursive invokation will end up processing the toplevel
> +Makefile.target file (more on that later).
> +
> +
> +- */Makefile.objs
> +
> +Since the source code is spread across multiple directories, the rules
> +for each file are similarly modularized. Thus each subdirectory
> +containing .c files will usually also contain a Makefile.objs file.
> +These files are not directly invoked by a recursive make, but instead
> +they are imported by the top level Makefile and/or Makefile.target
> +
> +Each Makefile.objs usually just declares a set of variables listing the
> +.o files that need building from the source files in the directory. They
> +will also define any custom linker or compiler flags. For example in
> +block/Makefile.objs
> +
> +  block-obj-$(CONFIG_LIBISCSI) += iscsi.o
> +  block-obj-$(CONFIG_CURL) += curl.o
> +
> +  ..snip...
> +
> +  iscsi.o-cflags     := $(LIBISCSI_CFLAGS)
> +  iscsi.o-libs       := $(LIBISCSI_LIBS)
> +  curl.o-cflags      := $(CURL_CFLAGS)
> +  curl.o-libs        := $(CURL_LIBS)
> +
> +
> +- Makefile.target
> +
> +This file provides the entry point used to build each individual system
> +or userspace emulator target. Each enabled target has its own
> +subdirectory. For example if configure is run with the argument
> +'--target-list=x86_64-softmmu', then a sub-directory 'x86_64-softmu'
> +will be created, containing a 'Makefile' which symlinks back to
> +Makefile.target
> +
> +So when the recursive '$(MAKE) -C x86_64-softmmu' is invoked, it ends up
> +using Makefile.target for the build rules.
> +
> +
> +- rules.mak
> +
> +This file provides the generic helper rules for invoking build tools, in
> +particular the compiler and linker. This also contains the magic (hairy)
> +'unnest-vars' function which is used to merge the variable definitions
> +from all Makefile.objs in the source tree down into the main Makefile
> +context.
> +
> +
> +- default-configs/*.mak
> +
> +The files under default-configs/ control what emulated hardware is built
> +into each QEMU system and userspace emulator targets. They merely
> +contain a long list of config variable definitions. For example,
> +default-configs/x86_64-softmmu.mak has:
> +
> +  include pci.mak
> +  include sound.mak
> +  include usb.mak
> +  CONFIG_QXL=$(CONFIG_SPICE)
> +  CONFIG_VGA_ISA=y
> +  CONFIG_VGA_CIRRUS=y
> +  CONFIG_VMWARE_VGA=y
> +  CONFIG_VIRTIO_VGA=y
> +  ...snip...
> +
> +These files rarely need changing unless new devices / hardware need to
> +be enabled for a particular system/userspace emulation target
> +
> +
> +- tests/Makefile
> +
> +Rules for building the unit tests. This file is included directly by the
> +top level Makefile, so anything defined in this file will influence the
> +entire build system. Care needs to be taken when writing rules for tests
> +to ensure they only apply to the unit test execution / build.
> +
> +
> +- po/Makefile
> +
> +Rules for building and installing the binary message catalogs from the
> +text .po file sources. This almost never needs changing for any reason.
> +
> +
> +Dynamically created files
> +-------------------------
> +
> +The following files are generated dynamically by configure in order to
> +control the behaviour of the staticaly defined makefiles. This avoids
> +the need for QEMU makefiles to go through any pre-processing as seen
> +with autotools, where Makefile.am generates Makefile.in which generates
> +Makefile.
> +
> +- config-host.mak
> +
> +When configure has determined the characteristics of the build host it
> +will write a long list of variables to config-host.mak file. This
> +provides the various install directories, compiler / linker flags and a
> +variable of CONFIG_* variables related to optionally enabled features.

ITYM:
- "variety of CONFIG_* variables", or
- "various CONFIG_* variables", and not
- "variable of CONFIG_* variables"

Suggestion wrt. meaning: can you stress that these CONFIG_* flags are
usable in the C source code as macros?

Also, for example, one macro I've found occasionally critical in the
past is "HOST_<ARCH>", which gives the host architecture (= uppercase
form of the "uname -m" output).

It doesn't match the CONFIG_* pattern, but I think it merits a separate
mention.

... Or maybe not; this is just a personal preference :)

> +This is imported by the top level Makefile in order to tailor the build
> +output.
> +
> +The variables defined here are those which are applicable to all QEMU
> +build outputs. Variables which are potentially different for each
> +emulator target are defined by the next file...
> +
> +It is also used as a dependancy checking mechanism. If make sees that

wiktionary calls this "archaic", so I'd suggest "dependency"

> +the modification timestamp on configure is newer than that on
> +config-host.mak, then configure will be re-run.
> +
> +
> +- $TARGET-NAME/config-target.mak
> +
> +TARGET-NAME is the name of a system or userspace emulator, for example,
> +x86_64-softmmu denotes the system emulator for the x86_64 architecture.
> +This file contains the variables which need to vary on a per-target
> +basis. For example, it will indicate whether KVM or Xen are enabled for
> +the target and any other potential custom libraries needed for linking
> +the target.
> +
> +
> +- $TARGET-NAME/config-devices.mak
> +
> +TARGET-NAME is again the name of a system or userspace emulator. The
> +config-devices.mak file is automatically generated by make using the
> +scripts/make_device_config.sh program, feeding it the
> +default-configs/$TARGET-NAME file as input.
> +
> +
> +- $TARGET-NAME/Makefile
> +
> +This is the entrypoint used when make recurses to build a single system
> +or userspace emulator target. It is merely a symlink back to the
> +Makefile.target in the top level.
> 

With the above suggestions (or without :)),

Acked-by: Laszlo Ersek <lersek@redhat.com>

(I'd really like this to be an R-b instead, but I don't feel I'm an
"expert" on this, so let me be modest. :))

Thanks!
Laszlo

On 09/22/2015 10:35 AM, Daniel P. Berrange wrote:
> Developers who are new to QEMU, or have a background familiarity
> with GNU autotools can have trouble getting their head around the

s/autotools/autotools,/

> home-grown QEMU build system. This document attempts to explain
> the structure / design of the configure script and the various
> Makefile pieces that live across the source tree.
> 
> Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
> ---
>  docs/build-system.txt | 493 ++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 493 insertions(+)
>  create mode 100644 docs/build-system.txt
> 

> +Stage 1: configure

> +
> +In contrast to autoconf scripts, QEMU's configure is expected to be
> +silent while it is checking for features. It will only display output
> +when an error occurrs, or to show the final feature enablement summary

s/occurrs/occurs/

> +on completion.
> +
> +Adding new checks to the configure script usually comprises the
> +following tasks
> +
> + - Initialize one or more variables with the default feature state.
> +
> +   Ideally features should auto-detect whether they are present,
> +   so try to avoid hardcoding the initial state to either enabled
> +   or disabled, as that forces the user to pass a --{dis,en}able-XXX
> +   flag on every invokation of configure

s/invokation/invocation/


> +Stage 2: makefiles
> +==================
> +
> +Although the source code is spread across multiple subdirectories, the
> +build system should be considered largely non-recursive in nature, in
> +contrast to common practices seen with automake. There is some recursive
> +invokation of make, but this is related to the things being built,
> +rather than the source directory structure.

s/invokation/invocation/

Mention that we require GNU make.

> +Module structure
> +----------------
> +
> +There are a number of key outputs of the QEMU build system
> +
> + - Tools - qemu-img, qemu-nbd, qga (guest agent), etc
> + - System emulators - qemu-system-$ARCH
> + - Userspace emulators - qemu-$ARCH
> + - Unit tests
> +
> +The source code is highly modularized, split across many files to
> +facilitate building of all of these components with as little duplicated
> +compilation as possible. There can be considered to be two distinct
> +groups of files, those which are independant of the QEMU emulation
> +target and those which are dependant on the QEMU emulation target.

Throughout this section: s/(in)?dependant/dependent/

> +
> +Statically defined files
> +------------------------

> +The recursive invokation will end up processing the toplevel

s/invokation/invocation/


> +
> +Dynamically created files
> +-------------------------
> +
> +The following files are generated dynamically by configure in order to
> +control the behaviour of the staticaly defined makefiles. This avoids

s/staticaly/statically/


> +
> +It is also used as a dependancy checking mechanism. If make sees that

s/dependancy/dependency/

meta review of your review:

On 09/22/15 20:11, John Snow wrote:
> Reviewed from an en_US perspective, though I left alone things that are
> clearly regional (e.g. 'behaviour' vs 'behavior')
> 
> On 09/22/2015 12:35 PM, Daniel P. Berrange wrote:

>> + - Add information to the help output message to report on the new
>> +   feature flag.
>> +
> 
> Remove period, or add to the other list items for consistency. My
> personal preference is to use the period for any sentences with proper
> grammatical structure, omitting it for simple list items.

Then:

>> +which create binaries must include the $(EXESUF) variable on the binary
>> +name. eg
> 
> 'e.g.' here and everywhere subsequent.

Self-contradiction found!!!1111eleven

:)

Honestly I'm surprised (or not) how many typos you've found that I
blissfully slid over.

>> +Each system/userspace emulation target needs to have a slightly
>> +different set of make rules / variables. Thus, make will be recursively
>> +invoked for each of the emulation targets.
>> +
>> +The recursive invokation will end up processing the toplevel
> 
> invocation again.

Self-contradictory period again! :)

> Thanks for writing this!

Yes!

> Pretending to be Eric,

Yes. :)

Cheers
Laszlo

On 09/22/2015 12:28 PM, Laszlo Ersek wrote:
> meta review of your review:
> 

> 
>>> +Each system/userspace emulation target needs to have a slightly
>>> +different set of make rules / variables. Thus, make will be recursively
>>> +invoked for each of the emulation targets.
>>> +
>>> +The recursive invokation will end up processing the toplevel
>>
>> invocation again.
> 
> Self-contradictory period again! :)
> 
>> Thanks for writing this!

Heartily seconded - in spite of our review comments, the document looks
very useful.

> 
> Yes!
> 
>> Pretending to be Eric,
> 
> Yes. :)

Don't know if I should be pleased or shocked :)

On 09/22/15 20:51, Eric Blake wrote:
> On 09/22/2015 12:28 PM, Laszlo Ersek wrote:
>> meta review of your review:
>>
> 
>>
>>>> +Each system/userspace emulation target needs to have a slightly
>>>> +different set of make rules / variables. Thus, make will be recursively
>>>> +invoked for each of the emulation targets.
>>>> +
>>>> +The recursive invokation will end up processing the toplevel
>>>
>>> invocation again.
>>
>> Self-contradictory period again! :)
>>
>>> Thanks for writing this!
> 
> Heartily seconded - in spite of our review comments, the document looks
> very useful.
> 
>>
>> Yes!
>>
>>> Pretending to be Eric,
>>
>> Yes. :)
> 
> Don't know if I should be pleased or shocked :)

Pleased, certainly. :)

Laszlo

On 22/09/2015 20:11, John Snow wrote:
>> +such as error handling infrastructure, standard data structures,
>> +platform portability wrapper functions, etc. This code can be compiled
>> +once only and the .o files linked into all output binaries.
>> +
>> +In the target dependant set lives CPU emulation, device emulation and
>> +much glue code. This code is generally compiled multiple times, once for

s/is generally compiled/sometimes also has to be compiled/

>> +each target architecture being built.

... while in other cases can be compiled once for each architecture.

Do not use "target architecture", since you're using "target" to mean a
Makefile target, i.e. binary.

>> +The target independant code that is used by all binaries is built into a

s/The target independant/Utlity/

>> +static archive called libqemuutil.a, which is then linked to all the
>> +binaries. Due to ongoing incomplete refactoring, some of the code in

s/Due to ongoing incomplete refactoring/In order to provide hooks that
are only needed by some of the binaries/

>> +libqemuutil.a depends on other functions that are not available in all

s/available in/fully implemented by/

>> +QEMU binaries. To deal with this there is a second library called
>> +libqemustub.a which provide dummy stubs for all these functions. These
>> +will get lazy linked into the binary if the real implementation is not
>> +present. Thus any time a binary links to libqemuutil.a, it should also
>> +be made to link libqemustub.a. eg
>> +
>> + qemu-img$(EXESUF): qemu-img.o ..snip.. libqemuutil.a libqemustub.a

Really all binaries should link libqemustub.a.

Perhaps add a note that libqemustub symbols effectively work as weak
symbols, but a static library is more portable?

>> +
>> +Windows platform portability
>> +----------------------------
>> +
>> +On Windows all binaries have a .exe suffix, so all the Makefile rules
> 
> I guess you pronounce the 'dot' :)
> 
>> +which create binaries must include the $(EXESUF) variable on the binary
>> +name. eg
> 
> 'e.g.' here and everywhere subsequent.
> 
>> +
>> + qemu-img$(EXESUF): qemu-img.o ..snip..
>> +
>> +This expands to '.exe' on Windows, or '' on other platforms.
>> +
>> +A further complication for the system and userspace emulator binaries is
>> +that two separate binaries need to be generated.
>> +
>> +The main binary (eg qemu-system-x86_64.exe) is linked against the
>> +Windows console runtime subsystem. These are expected to be run from a
>> +command prompt window, and so will print stderr to the console that
>> +launched them.
>> +
>> +The second binary generated has a 'w' on the end of its name (eg
>> +qemu-system-x86_64w.exe) and is linked against the Windows graphical
>> +runtime subsystem. These are expected to be run directly from the
>> +desktop and will open up a dedicated console window for stderr output.
>> +
>> +The Makefile.target will generate the binary for the graphical subsystem
>> +first, and then use objcopy to relink it against the console subsystem
>> +to generate the second binary.
>> +
>> +
>> +Object variable naming
>> +----------------------
>> +
>> +The QEMU convention is to define variables to list different groups of
>> +object files. These are named with the convention $PREFIX-y. For example

$PREFIX-obj-y

>> +the libqemuutil.a file will be linked with all objects listed in a
>> +variable 'util-y'.

util-obj-y.

>> +- */Makefile.objs
>> +
>> +Since the source code is spread across multiple directories, the rules
>> +for each file are similarly modularized. Thus each subdirectory
>> +containing .c files will usually also contain a Makefile.objs file.
>> +These files are not directly invoked by a recursive make, but instead
>> +they are imported by the top level Makefile and/or Makefile.target
>> +
>> +Each Makefile.objs usually just declares a set of variables listing the
>> +.o files that need building from the source files in the directory. They
>> +will also define any custom linker or compiler flags. For example in
>> +block/Makefile.objs
>> +
>> +  block-obj-$(CONFIG_LIBISCSI) += iscsi.o
>> +  block-obj-$(CONFIG_CURL) += curl.o
>> +
>> +  ..snip...
>> +
>> +  iscsi.o-cflags     := $(LIBISCSI_CFLAGS)
>> +  iscsi.o-libs       := $(LIBISCSI_LIBS)
>> +  curl.o-cflags      := $(CURL_CFLAGS)
>> +  curl.o-libs        := $(CURL_LIBS)

You may want to mention that rules in */Makefile.objs should use $(obj)
as a prefix to the target, for example:

$(obj)/generated-tcg-tracers.h: $(obj)/generated-tcg-tracers.h-timestamp

I'll join the choir: awesome job.

Paolo

On Tue, Sep 22, 2015 at 05:35:59PM +0100, Daniel P. Berrange wrote:
> Developers who are new to QEMU, or have a background familiarity
> with GNU autotools can have trouble getting their head around the
> home-grown QEMU build system. This document attempts to explain
> the structure / design of the configure script and the various
> Makefile pieces that live across the source tree.
> 
> Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
> ---
>  docs/build-system.txt | 493 ++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 493 insertions(+)
>  create mode 100644 docs/build-system.txt

Thanks for all the feedback on this. I won't reply to all the mails
individually, I'll simply address all the feedback given and post a
v2 later.

Regards,
Daniel