| Message ID | 20210219002605.1166831-3-xnox@ubuntu.com |
|---|---|
| State | New |
| Headers | show |
| Series | Implement noudeb build profile | expand |
On Fri, Feb 19, 2021 at 12:26:05AM +0000, Dimitri John Ledkov wrote: > When building under either of stage1, noudeb, cross or autopkgtest > profiles udeb packages are not desired. Stop generating them, and drop > build-dependencies that are used just to create udebs. > > Also note that dpkg in hirsute and up automatically buidlds packages > with a noudeb profile, thus hirsute+ builds will be now much faster. > > This is a correct and backwards compatibles support for build profiles > and can be backported to any prior release, where by default packages > will be build with udebs. > > BugLink: https://bugs.launchpad.net/bugs/1916095 > Signed-off-by: Dimitri John Ledkov <xnox@ubuntu.com> > --- > debian.master/control.d/flavour-control.stub | 2 +- > debian.master/control.stub.in | 4 ++-- > debian/rules | 5 +++++ > 3 files changed, 8 insertions(+), 3 deletions(-) > > diff --git a/debian.master/control.d/flavour-control.stub b/debian.master/control.d/flavour-control.stub > index 17be49dbd..df6377ce1 100644 > --- a/debian.master/control.d/flavour-control.stub > +++ b/debian.master/control.d/flavour-control.stub > @@ -141,7 +141,7 @@ Description: Linux kernel version specific cloud tools for version PKGVER-ABINUM > =HUMAN=. > > Package: SRCPKGNAME-udebs-FLAVOUR > -Build-Profiles: <!stage1> > +Build-Profiles: <!stage1> <!noudeb> <!cross> <!autopkgtest> > XC-Package-Type: udeb > Section: debian-installer > Architecture: ARCH > diff --git a/debian.master/control.stub.in b/debian.master/control.stub.in > index cae6f6783..1c1ee8857 100644 > --- a/debian.master/control.stub.in > +++ b/debian.master/control.stub.in > @@ -7,8 +7,8 @@ Build-Depends: > debhelper-compat (= 10), > dh-systemd, > cpio, > - kernel-wedge, > - dctrl-tools, > + kernel-wedge <!stage1> <!noudeb> <!cross> <!autopkgtest>, > + dctrl-tools <!stage1> <!noudeb> <!cross> <!autopkgtest>, > kmod <!stage1>, > makedumpfile [amd64] <!stage1>, > libcap-dev <!stage1>, > diff --git a/debian/rules b/debian/rules > index ceffe36be..bedcd4f60 100755 > --- a/debian/rules > +++ b/debian/rules > @@ -39,6 +39,9 @@ do_tools_common?=true > do_tools_host?=false > do_tools_perf_jvmti?=false > do_enforce_all?=false > +ifneq ($(filter noudeb,$(DEB_BUILD_PROFILES)),) > +disable_d_i=true > +endif I'm not that knowledgable about build profiles, so maybe this is a dumb question. There's a larger set of build profiles where you omit the build dependencies for the udebs, but you only disable the code which requires those tools when noudeb is in DEB_BUILD_PROFILES. Do the other profiles imply noudeb, such that DEB_BUILD_PROFILES will contain noudeb when those other build profiles are activated? Thanks, Seth > > # Don't build tools or udebs in a cross compile environment. > ifneq ($(DEB_HOST_ARCH),$(DEB_BUILD_ARCH)) > @@ -220,6 +223,7 @@ $(DEBIAN)/control.stub: \ > debian/control: $(DEBIAN)/control.stub > echo "# placebo control.stub for kernel-wedge flow change" >debian/control.stub > cp $(DEBIAN)/control.stub debian/control > +ifeq ($(disable_d_i),) > # start new paragraph in debian/control > echo >> debian/control > # append udeb packages > @@ -228,6 +232,7 @@ debian/control: $(DEBIAN)/control.stub > LANG=C kernel-wedge gen-control $(release)-$(abinum) | \ > grep-dctrl -FArchitecture $(arch) \ > >>$(CURDIR)/debian/control > +endif > > debian/canonical-certs.pem: $(wildcard $(DROOT)/certs/*-all.pem) $(wildcard $(DROOT)/certs/*-$(arch).pem) $(wildcard $(DEBIAN)/certs/*-all.pem) $(wildcard $(DEBIAN)/certs/*-$(arch).pem) > for cert in $(sort $(notdir $^)); \ > -- > 2.27.0 > > > -- > kernel-team mailing list > kernel-team@lists.ubuntu.com > https://lists.ubuntu.com/mailman/listinfo/kernel-team
On Fri, 19 Feb 2021 at 16:55, Seth Forshee <seth.forshee@canonical.com> wrote: > > On Fri, Feb 19, 2021 at 12:26:05AM +0000, Dimitri John Ledkov wrote: > > When building under either of stage1, noudeb, cross or autopkgtest > > profiles udeb packages are not desired. Stop generating them, and drop > > build-dependencies that are used just to create udebs. > > > > Also note that dpkg in hirsute and up automatically buidlds packages > > with a noudeb profile, thus hirsute+ builds will be now much faster. > > > > This is a correct and backwards compatibles support for build profiles > > and can be backported to any prior release, where by default packages > > will be build with udebs. > > > > BugLink: https://bugs.launchpad.net/bugs/1916095 > > Signed-off-by: Dimitri John Ledkov <xnox@ubuntu.com> > > --- > > debian.master/control.d/flavour-control.stub | 2 +- > > debian.master/control.stub.in | 4 ++-- > > debian/rules | 5 +++++ > > 3 files changed, 8 insertions(+), 3 deletions(-) > > > > diff --git a/debian.master/control.d/flavour-control.stub b/debian.master/control.d/flavour-control.stub > > index 17be49dbd..df6377ce1 100644 > > --- a/debian.master/control.d/flavour-control.stub > > +++ b/debian.master/control.d/flavour-control.stub > > @@ -141,7 +141,7 @@ Description: Linux kernel version specific cloud tools for version PKGVER-ABINUM > > =HUMAN=. > > > > Package: SRCPKGNAME-udebs-FLAVOUR > > -Build-Profiles: <!stage1> > > +Build-Profiles: <!stage1> <!noudeb> <!cross> <!autopkgtest> > > XC-Package-Type: udeb > > Section: debian-installer > > Architecture: ARCH > > diff --git a/debian.master/control.stub.in b/debian.master/control.stub.in > > index cae6f6783..1c1ee8857 100644 > > --- a/debian.master/control.stub.in > > +++ b/debian.master/control.stub.in > > @@ -7,8 +7,8 @@ Build-Depends: > > debhelper-compat (= 10), > > dh-systemd, > > cpio, > > - kernel-wedge, > > - dctrl-tools, > > + kernel-wedge <!stage1> <!noudeb> <!cross> <!autopkgtest>, > > + dctrl-tools <!stage1> <!noudeb> <!cross> <!autopkgtest>, > > kmod <!stage1>, > > makedumpfile [amd64] <!stage1>, > > libcap-dev <!stage1>, > > diff --git a/debian/rules b/debian/rules > > index ceffe36be..bedcd4f60 100755 > > --- a/debian/rules > > +++ b/debian/rules > > @@ -39,6 +39,9 @@ do_tools_common?=true > > do_tools_host?=false > > do_tools_perf_jvmti?=false > > do_enforce_all?=false > > +ifneq ($(filter noudeb,$(DEB_BUILD_PROFILES)),) > > +disable_d_i=true > > +endif > > I'm not that knowledgable about build profiles, so maybe this is a dumb > question. There's a larger set of build profiles where you omit the > build dependencies for the udebs, but you only disable the code which > requires those tools when noudeb is in DEB_BUILD_PROFILES. Do the other > profiles imply noudeb, such that DEB_BUILD_PROFILES will contain noudeb > when those other build profiles are activated? > Note that debian/control uses many profiles, but debian/rules mostly uses its own variables to decide what to do. Hence disable_d_i= <empty> means that debian/rules will be attempting to generate udebs and disable_d_i=true (or actually any value) will disable udebs. And there are many other hunks of code in debian/rules that set disable_d_i=true for cases of cross compilations (aka <cross>) and autopkgtest (aka <autopkgtest>). <cross> is not checked for via DEB_BUILD_PROFILES which older sbuild did not set, but by checking if BUILD & HOST arches are different, which should always be true under <cross> profile. autopkgtest profile is checked elsewhere and sets disable_d_i among other variables. To quickly observe that you can do $ git grep -C 3 disable_d_i debian There are mentions of "bootstrap" and "stage1" build profiles in the debian/rules, but i'm not too sure if those are actively used or maintained. So I tried not to make stage1 any worse than it is today. Maybe we should drop "bootstrap" and "stage1" builds. In Debian, one can use those to automatically cross build & bootstrap a brand new architecture from scratch with automatic stages progressing to build headers, toolchain, buildd chroot, native headers, kernel, etc. As one very long, but fully automated process. It is however mostly academic work, given that we don't rebootstrap from scratch when we open new distro series. > Thanks, > Seth > > > > > # Don't build tools or udebs in a cross compile environment. > > ifneq ($(DEB_HOST_ARCH),$(DEB_BUILD_ARCH)) > > @@ -220,6 +223,7 @@ $(DEBIAN)/control.stub: \ > > debian/control: $(DEBIAN)/control.stub > > echo "# placebo control.stub for kernel-wedge flow change" >debian/control.stub > > cp $(DEBIAN)/control.stub debian/control > > +ifeq ($(disable_d_i),) > > # start new paragraph in debian/control > > echo >> debian/control > > # append udeb packages > > @@ -228,6 +232,7 @@ debian/control: $(DEBIAN)/control.stub > > LANG=C kernel-wedge gen-control $(release)-$(abinum) | \ > > grep-dctrl -FArchitecture $(arch) \ > > >>$(CURDIR)/debian/control > > +endif > > > > debian/canonical-certs.pem: $(wildcard $(DROOT)/certs/*-all.pem) $(wildcard $(DROOT)/certs/*-$(arch).pem) $(wildcard $(DEBIAN)/certs/*-all.pem) $(wildcard $(DEBIAN)/certs/*-$(arch).pem) > > for cert in $(sort $(notdir $^)); \ > > -- > > 2.27.0 > > > > > > -- > > kernel-team mailing list > > kernel-team@lists.ubuntu.com > > https://lists.ubuntu.com/mailman/listinfo/kernel-team
On Sat, Feb 20, 2021 at 02:26:25AM +0000, Dimitri John Ledkov wrote: > On Fri, 19 Feb 2021 at 16:55, Seth Forshee <seth.forshee@canonical.com> wrote: > > > > On Fri, Feb 19, 2021 at 12:26:05AM +0000, Dimitri John Ledkov wrote: > > > When building under either of stage1, noudeb, cross or autopkgtest > > > profiles udeb packages are not desired. Stop generating them, and drop > > > build-dependencies that are used just to create udebs. > > > > > > Also note that dpkg in hirsute and up automatically buidlds packages > > > with a noudeb profile, thus hirsute+ builds will be now much faster. > > > > > > This is a correct and backwards compatibles support for build profiles > > > and can be backported to any prior release, where by default packages > > > will be build with udebs. > > > > > > BugLink: https://bugs.launchpad.net/bugs/1916095 > > > Signed-off-by: Dimitri John Ledkov <xnox@ubuntu.com> > > > --- > > > debian.master/control.d/flavour-control.stub | 2 +- > > > debian.master/control.stub.in | 4 ++-- > > > debian/rules | 5 +++++ > > > 3 files changed, 8 insertions(+), 3 deletions(-) > > > > > > diff --git a/debian.master/control.d/flavour-control.stub b/debian.master/control.d/flavour-control.stub > > > index 17be49dbd..df6377ce1 100644 > > > --- a/debian.master/control.d/flavour-control.stub > > > +++ b/debian.master/control.d/flavour-control.stub > > > @@ -141,7 +141,7 @@ Description: Linux kernel version specific cloud tools for version PKGVER-ABINUM > > > =HUMAN=. > > > > > > Package: SRCPKGNAME-udebs-FLAVOUR > > > -Build-Profiles: <!stage1> > > > +Build-Profiles: <!stage1> <!noudeb> <!cross> <!autopkgtest> > > > XC-Package-Type: udeb > > > Section: debian-installer > > > Architecture: ARCH > > > diff --git a/debian.master/control.stub.in b/debian.master/control.stub.in > > > index cae6f6783..1c1ee8857 100644 > > > --- a/debian.master/control.stub.in > > > +++ b/debian.master/control.stub.in > > > @@ -7,8 +7,8 @@ Build-Depends: > > > debhelper-compat (= 10), > > > dh-systemd, > > > cpio, > > > - kernel-wedge, > > > - dctrl-tools, > > > + kernel-wedge <!stage1> <!noudeb> <!cross> <!autopkgtest>, > > > + dctrl-tools <!stage1> <!noudeb> <!cross> <!autopkgtest>, > > > kmod <!stage1>, > > > makedumpfile [amd64] <!stage1>, > > > libcap-dev <!stage1>, > > > diff --git a/debian/rules b/debian/rules > > > index ceffe36be..bedcd4f60 100755 > > > --- a/debian/rules > > > +++ b/debian/rules > > > @@ -39,6 +39,9 @@ do_tools_common?=true > > > do_tools_host?=false > > > do_tools_perf_jvmti?=false > > > do_enforce_all?=false > > > +ifneq ($(filter noudeb,$(DEB_BUILD_PROFILES)),) > > > +disable_d_i=true > > > +endif > > > > I'm not that knowledgable about build profiles, so maybe this is a dumb > > question. There's a larger set of build profiles where you omit the > > build dependencies for the udebs, but you only disable the code which > > requires those tools when noudeb is in DEB_BUILD_PROFILES. Do the other > > profiles imply noudeb, such that DEB_BUILD_PROFILES will contain noudeb > > when those other build profiles are activated? > > > > Note that debian/control uses many profiles, but debian/rules mostly > uses its own variables to decide what to do. > Hence disable_d_i= <empty> means that debian/rules will be attempting > to generate udebs and disable_d_i=true (or actually any value) will > disable udebs. > > And there are many other hunks of code in debian/rules that set > disable_d_i=true for cases of cross compilations (aka <cross>) and > autopkgtest (aka <autopkgtest>). > > <cross> is not checked for via DEB_BUILD_PROFILES which older sbuild > did not set, but by checking if BUILD & HOST arches are different, > which should always be true under <cross> profile. > > autopkgtest profile is checked elsewhere and sets disable_d_i among > other variables. > > To quickly observe that you can do $ git grep -C 3 disable_d_i debian > > There are mentions of "bootstrap" and "stage1" build profiles in the > debian/rules, but i'm not too sure if those are actively used or > maintained. So I tried not to make stage1 any worse than it is today. > Maybe we should drop "bootstrap" and "stage1" builds. In Debian, one > can use those to automatically cross build & bootstrap a brand new > architecture from scratch with automatic stages progressing to build > headers, toolchain, buildd chroot, native headers, kernel, etc. As one > very long, but fully automated process. It is however mostly academic > work, given that we don't rebootstrap from scratch when we open new > distro series. Thanks for the explanation. Sounds like this shouldn't break anything for at least our typical workflows, and speeding up our builds is very welcome. Acked-by: Seth Forshee <seth.forshee@canonical.com> > > > Thanks, > > Seth > > > > > > > > # Don't build tools or udebs in a cross compile environment. > > > ifneq ($(DEB_HOST_ARCH),$(DEB_BUILD_ARCH)) > > > @@ -220,6 +223,7 @@ $(DEBIAN)/control.stub: \ > > > debian/control: $(DEBIAN)/control.stub > > > echo "# placebo control.stub for kernel-wedge flow change" >debian/control.stub > > > cp $(DEBIAN)/control.stub debian/control > > > +ifeq ($(disable_d_i),) > > > # start new paragraph in debian/control > > > echo >> debian/control > > > # append udeb packages > > > @@ -228,6 +232,7 @@ debian/control: $(DEBIAN)/control.stub > > > LANG=C kernel-wedge gen-control $(release)-$(abinum) | \ > > > grep-dctrl -FArchitecture $(arch) \ > > > >>$(CURDIR)/debian/control > > > +endif > > > > > > debian/canonical-certs.pem: $(wildcard $(DROOT)/certs/*-all.pem) $(wildcard $(DROOT)/certs/*-$(arch).pem) $(wildcard $(DEBIAN)/certs/*-all.pem) $(wildcard $(DEBIAN)/certs/*-$(arch).pem) > > > for cert in $(sort $(notdir $^)); \ > > > -- > > > 2.27.0 > > > > > > > > > -- > > > kernel-team mailing list > > > kernel-team@lists.ubuntu.com > > > https://lists.ubuntu.com/mailman/listinfo/kernel-team > > > > -- > Regards, > > Dimitri.
diff --git a/debian.master/control.d/flavour-control.stub b/debian.master/control.d/flavour-control.stub index 17be49dbd..df6377ce1 100644 --- a/debian.master/control.d/flavour-control.stub +++ b/debian.master/control.d/flavour-control.stub @@ -141,7 +141,7 @@ Description: Linux kernel version specific cloud tools for version PKGVER-ABINUM =HUMAN=. Package: SRCPKGNAME-udebs-FLAVOUR -Build-Profiles: <!stage1> +Build-Profiles: <!stage1> <!noudeb> <!cross> <!autopkgtest> XC-Package-Type: udeb Section: debian-installer Architecture: ARCH diff --git a/debian.master/control.stub.in b/debian.master/control.stub.in index cae6f6783..1c1ee8857 100644 --- a/debian.master/control.stub.in +++ b/debian.master/control.stub.in @@ -7,8 +7,8 @@ Build-Depends: debhelper-compat (= 10), dh-systemd, cpio, - kernel-wedge, - dctrl-tools, + kernel-wedge <!stage1> <!noudeb> <!cross> <!autopkgtest>, + dctrl-tools <!stage1> <!noudeb> <!cross> <!autopkgtest>, kmod <!stage1>, makedumpfile [amd64] <!stage1>, libcap-dev <!stage1>, diff --git a/debian/rules b/debian/rules index ceffe36be..bedcd4f60 100755 --- a/debian/rules +++ b/debian/rules @@ -39,6 +39,9 @@ do_tools_common?=true do_tools_host?=false do_tools_perf_jvmti?=false do_enforce_all?=false +ifneq ($(filter noudeb,$(DEB_BUILD_PROFILES)),) +disable_d_i=true +endif # Don't build tools or udebs in a cross compile environment. ifneq ($(DEB_HOST_ARCH),$(DEB_BUILD_ARCH)) @@ -220,6 +223,7 @@ $(DEBIAN)/control.stub: \ debian/control: $(DEBIAN)/control.stub echo "# placebo control.stub for kernel-wedge flow change" >debian/control.stub cp $(DEBIAN)/control.stub debian/control +ifeq ($(disable_d_i),) # start new paragraph in debian/control echo >> debian/control # append udeb packages @@ -228,6 +232,7 @@ debian/control: $(DEBIAN)/control.stub LANG=C kernel-wedge gen-control $(release)-$(abinum) | \ grep-dctrl -FArchitecture $(arch) \ >>$(CURDIR)/debian/control +endif debian/canonical-certs.pem: $(wildcard $(DROOT)/certs/*-all.pem) $(wildcard $(DROOT)/certs/*-$(arch).pem) $(wildcard $(DEBIAN)/certs/*-all.pem) $(wildcard $(DEBIAN)/certs/*-$(arch).pem) for cert in $(sort $(notdir $^)); \
When building under either of stage1, noudeb, cross or autopkgtest profiles udeb packages are not desired. Stop generating them, and drop build-dependencies that are used just to create udebs. Also note that dpkg in hirsute and up automatically buidlds packages with a noudeb profile, thus hirsute+ builds will be now much faster. This is a correct and backwards compatibles support for build profiles and can be backported to any prior release, where by default packages will be build with udebs. BugLink: https://bugs.launchpad.net/bugs/1916095 Signed-off-by: Dimitri John Ledkov <xnox@ubuntu.com> --- debian.master/control.d/flavour-control.stub | 2 +- debian.master/control.stub.in | 4 ++-- debian/rules | 5 +++++ 3 files changed, 8 insertions(+), 3 deletions(-)