[0/6,Unstable] Kernel package builds running out of space on builders

BugLink: https://bugs.launchpad.net/bugs/1930713

[Impact]

Our kernel builds are sometimes running out of space on the builders when we
are building multiple flavours. We've seen this with focal:linux-hwe for amd64
and impish:linux-unstable for arm64. This is in part because package builds are
broken up into a build phase (which builds the source tree) and a binary phase
(which creates the debs). These are run separately, with the binary phase run
under fakeroot to get correct ownership for files in the package archives,
requiring builds for multiple flavours to be present on disk at the same time.

We have implemented various fixes for this problem over time, and explored many
others which have not worked out. But the size of the kernel keeps increasing,
and now it seems our only remaining option is to build one flavour and install
its files, then remove the flavour build files before building the next
flavour. This means that files are installed for later package builds during
the build phase, requiring that ownership of these files be fixed up during the
binary phase to get correct ownership in the package archives.

[Test Plan]

Build a full set of kernel packages (inlcuding linux-source and dbgsym
packages, which are generally excluded when not building on builders) at a
given tag, then build another set from the same tag with the packaging changes
applied. Compare the resulting debs to confirm that the set of produced
packages is the same, the file lists within the packages are the same, and that
file ownership and permissions between the packages is identical.

I have done this testing with the proposed patches with a recent linux-unstable
tag and found no differences with and without the changes.

[Where problems could occur]

Incorrect ownership of files in the package archives is the main concern. I
have tested for this, but it is possible that future upstream changes could
unexpectedly result in files with incorrect ownership.

Reordering of the package build sequence could result in missing files which
should be in packages, or files present in packages which should be excluded.
Some instances of this occurred while developing these changes and have been
fixed. Future updates to upstream or to the packaging could cause additional
issues.

Changes since v1:
 - Test paths to decide whether to fix ownership for a package instead of
   using build variables.

Thanks,
Seth

Seth Forshee (6):
  UBUNTU: [Debian] use stamps for flavour install targets
  UBUNTU: [Debian] run install-$(flavour) targets during build phase
  UBUNTU: [Debian] remove dh_testroot from install targets
  UBUNTU: [Debian] dkms-build -- use fakeroot if not running as root
  UBUNTU: [Debian] exclude $(DEBIAN)/__abi.current from linux-source
  UBUNTU: [Debian] fix ownership for files installed during build

 debian/rules.d/2-binary-arch.mk  | 76 +++++++++++++++++++++-----------
 debian/rules.d/3-binary-indep.mk | 18 ++++----
 debian/rules.d/4-checks.mk       |  6 +--
 debian/scripts/dkms-build        |  4 +-
 4 files changed, 66 insertions(+), 38 deletions(-)

On Thu, Jun 03, 2021 at 09:47:13PM -0500, Seth Forshee wrote:
> BugLink: https://bugs.launchpad.net/bugs/1930713
> 
> [Impact]
> 
> Our kernel builds are sometimes running out of space on the builders when we
> are building multiple flavours. We've seen this with focal:linux-hwe for amd64
> and impish:linux-unstable for arm64. This is in part because package builds are
> broken up into a build phase (which builds the source tree) and a binary phase
> (which creates the debs). These are run separately, with the binary phase run
> under fakeroot to get correct ownership for files in the package archives,
> requiring builds for multiple flavours to be present on disk at the same time.
> 
> We have implemented various fixes for this problem over time, and explored many
> others which have not worked out. But the size of the kernel keeps increasing,
> and now it seems our only remaining option is to build one flavour and install
> its files, then remove the flavour build files before building the next
> flavour. This means that files are installed for later package builds during
> the build phase, requiring that ownership of these files be fixed up during the
> binary phase to get correct ownership in the package archives.
> 
> [Test Plan]
> 
> Build a full set of kernel packages (inlcuding linux-source and dbgsym
> packages, which are generally excluded when not building on builders) at a
> given tag, then build another set from the same tag with the packaging changes
> applied. Compare the resulting debs to confirm that the set of produced
> packages is the same, the file lists within the packages are the same, and that
> file ownership and permissions between the packages is identical.
> 
> I have done this testing with the proposed patches with a recent linux-unstable
> tag and found no differences with and without the changes.
> 
> [Where problems could occur]
> 
> Incorrect ownership of files in the package archives is the main concern. I
> have tested for this, but it is possible that future upstream changes could
> unexpectedly result in files with incorrect ownership.
> 
> Reordering of the package build sequence could result in missing files which
> should be in packages, or files present in packages which should be excluded.
> Some instances of this occurred while developing these changes and have been
> fixed. Future updates to upstream or to the packaging could cause additional
> issues.
> 
> Changes since v1:
>  - Test paths to decide whether to fix ownership for a package instead of
>    using build variables.

Applied patches 1-5 to unstable/master.