| Message ID | 4E5416DC.90205@canonical.com |
|---|---|
| State | New |
| Headers | show |
On Tue, Aug 23, 2011 at 03:08:44PM -0600, Tim Gardner wrote: > These patches in Oneiric seem to be working pretty well. With these > changes and sufficient server capability I've seen build times cut > in half. Now you'll be able to control concurrency from the command > line, e.g., > > dpkg-buildpackage -j`getconf _NPROCESSORS_ONLN` -B > > http://bugs.launchpad.net/bugs/832352 > > The following changes since commit 2f28aed962765c698f9ed3bc2385dbe79ada2514: > Seth Forshee (1): > drm/i915: Remove BUG_ON from i915_gem_evict_something > > are available in the git repository at: > > git://kernel.ubuntu.com/rtg/ubuntu-lucid.git makefile-improvenets > > Tim Gardner (11): > UBUNTU: [Config] Simplify binary-udebs dependencies > UBUNTU: [Config] kernel preparation cannot be parallelized > UBUNTU: [Config] Linearize module/abi checks > UBUNTU: [Config] Linearize and simplify tree preparation rules > UBUNTU: [Config] Build kernel image in parallel with modules > UBUNTU: [Config] Set concurrency for kmake invocations > UBUNTU: [Config] Improve install-arch-headers speed > UBUNTU: [Config] Fix binary-perarch dependencies > UBUNTU: [Config] Removed stamp-flavours target > UBUNTU: [Config] Serialize binary indep targets > UBUNTU: [Config] Use build stamp directly I'm no make expert, but I noted some things. In changes UBUNTU: [Config] kernel preparation cannot be parallelized UBUNTU: [Config] Set concurrency for kmake invocations you force -j1. Indeed scripts target depend on oldconfig/prepare first: $(build_cd) $(kmake) $(build_O) -j1 silentoldconfig prepare scripts but each target could benefit from parallelism (which I expect to be the case of scripts target), so a better invocation would be: $(build_cd) $(kmake) $(build_O) silentoldconfig $(build_cd) $(kmake) $(build_O) prepare $(build_cd) $(kmake) $(build_O) scripts Also, prepare alread implies/runs silentoldconfig, so we could simplify even more I think: $(build_cd) $(kmake) $(build_O) prepare $(build_cd) $(kmake) $(build_O) scripts I don't expect significant improvement, but would be better. In change UBUNTU: [Config] Linearize module/abi checks you could remove the install -d below "+$(abidir)/%.modules: abi-check-%" since through abi-check-% -> $(abidir)/% we already get the directory created. Also, on a quick look, seems this change is unecessary: abi-check script doesn't seem change/care about the modules list module-check script also only cares about modules list Unless I'm missing something else. In change UBUNTU: [Config] Serialize binary indep targets I'm in doubt, why serialize it? The rest looks ok to me, it's great having a better build time. > > debian/rules.d/2-binary-arch.mk | 35 > ++++++++++++++--------------------- > debian/rules.d/3-binary-indep.mk | 29 ++++++++++++----------------- > debian/rules.d/4-checks.mk | 10 ++-------- > debian/rules.d/5-udebs.mk | 4 ++-- > 4 files changed, 30 insertions(+), 48 deletions(-) > -- > Tim Gardner tim.gardner@canonical.com > > -- > kernel-team mailing list > kernel-team@lists.ubuntu.com > https://lists.ubuntu.com/mailman/listinfo/kernel-team >
On Thu, Aug 25, 2011 at 05:36:31PM -0300, Herton Ronaldo Krzesinski wrote: > On Tue, Aug 23, 2011 at 03:08:44PM -0600, Tim Gardner wrote: > > These patches in Oneiric seem to be working pretty well. With these > > changes and sufficient server capability I've seen build times cut > > in half. Now you'll be able to control concurrency from the command > > line, e.g., > > > > dpkg-buildpackage -j`getconf _NPROCESSORS_ONLN` -B > > > > http://bugs.launchpad.net/bugs/832352 > > > > The following changes since commit 2f28aed962765c698f9ed3bc2385dbe79ada2514: > > Seth Forshee (1): > > drm/i915: Remove BUG_ON from i915_gem_evict_something > > > > are available in the git repository at: > > > > git://kernel.ubuntu.com/rtg/ubuntu-lucid.git makefile-improvenets > > > > Tim Gardner (11): > > UBUNTU: [Config] Simplify binary-udebs dependencies > > UBUNTU: [Config] kernel preparation cannot be parallelized > > UBUNTU: [Config] Linearize module/abi checks > > UBUNTU: [Config] Linearize and simplify tree preparation rules > > UBUNTU: [Config] Build kernel image in parallel with modules > > UBUNTU: [Config] Set concurrency for kmake invocations > > UBUNTU: [Config] Improve install-arch-headers speed > > UBUNTU: [Config] Fix binary-perarch dependencies > > UBUNTU: [Config] Removed stamp-flavours target > > UBUNTU: [Config] Serialize binary indep targets > > UBUNTU: [Config] Use build stamp directly > > I'm no make expert, but I noted some things. > > In changes > > UBUNTU: [Config] kernel preparation cannot be parallelized > UBUNTU: [Config] Set concurrency for kmake invocations > > you force -j1. Indeed scripts target depend on oldconfig/prepare first: > > $(build_cd) $(kmake) $(build_O) -j1 silentoldconfig prepare scripts > > but each target could benefit from parallelism (which I expect to be the > case of scripts target), so a better invocation would be: > > $(build_cd) $(kmake) $(build_O) silentoldconfig > $(build_cd) $(kmake) $(build_O) prepare > $(build_cd) $(kmake) $(build_O) scripts > > Also, prepare alread implies/runs silentoldconfig, so we could simplify > even more I think: > > $(build_cd) $(kmake) $(build_O) prepare > $(build_cd) $(kmake) $(build_O) scripts I missed $(conc_level), should be: $(build_cd) $(kmake) $(build_O) $(conc_level) prepare $(build_cd) $(kmake) $(build_O) $(conc_level) scripts > > I don't expect significant improvement, but would be better. > > In change > > UBUNTU: [Config] Linearize module/abi checks > > you could remove the install -d below "+$(abidir)/%.modules: abi-check-%" > since through abi-check-% -> $(abidir)/% we already get the directory > created. > > Also, on a quick look, seems this change is unecessary: > abi-check script doesn't seem change/care about the modules list > module-check script also only cares about modules list > Unless I'm missing something else. > > > In change > > UBUNTU: [Config] Serialize binary indep targets > > I'm in doubt, why serialize it? > > > The rest looks ok to me, it's great having a better build time. > > > > > debian/rules.d/2-binary-arch.mk | 35 > > ++++++++++++++--------------------- > > debian/rules.d/3-binary-indep.mk | 29 ++++++++++++----------------- > > debian/rules.d/4-checks.mk | 10 ++-------- > > debian/rules.d/5-udebs.mk | 4 ++-- > > 4 files changed, 30 insertions(+), 48 deletions(-) > > -- > > Tim Gardner tim.gardner@canonical.com > > > > -- > > kernel-team mailing list > > kernel-team@lists.ubuntu.com > > https://lists.ubuntu.com/mailman/listinfo/kernel-team > > > > -- > []'s > Herton > > -- > kernel-team mailing list > kernel-team@lists.ubuntu.com > https://lists.ubuntu.com/mailman/listinfo/kernel-team >
On 23.08.2011 23:08, Tim Gardner wrote: > These patches in Oneiric seem to be working pretty well. With these changes and > sufficient server capability I've seen build times cut in half. Now you'll be > able to control concurrency from the command line, e.g., > > dpkg-buildpackage -j`getconf _NPROCESSORS_ONLN` -B > > http://bugs.launchpad.net/bugs/832352 > > The following changes since commit 2f28aed962765c698f9ed3bc2385dbe79ada2514: > Seth Forshee (1): > drm/i915: Remove BUG_ON from i915_gem_evict_something > > are available in the git repository at: > > git://kernel.ubuntu.com/rtg/ubuntu-lucid.git makefile-improvenets > > Tim Gardner (11): > UBUNTU: [Config] Simplify binary-udebs dependencies > UBUNTU: [Config] kernel preparation cannot be parallelized > UBUNTU: [Config] Linearize module/abi checks > UBUNTU: [Config] Linearize and simplify tree preparation rules > UBUNTU: [Config] Build kernel image in parallel with modules > UBUNTU: [Config] Set concurrency for kmake invocations > UBUNTU: [Config] Improve install-arch-headers speed > UBUNTU: [Config] Fix binary-perarch dependencies > UBUNTU: [Config] Removed stamp-flavours target > UBUNTU: [Config] Serialize binary indep targets > UBUNTU: [Config] Use build stamp directly > > debian/rules.d/2-binary-arch.mk | 35 ++++++++++++++--------------------- > debian/rules.d/3-binary-indep.mk | 29 ++++++++++++----------------- > debian/rules.d/4-checks.mk | 10 ++-------- > debian/rules.d/5-udebs.mk | 4 ++-- > 4 files changed, 30 insertions(+), 48 deletions(-) As much as I would like improved build times, the voice on the other side tells me it may be just a good idea to let changes/improvements settle a bit in Oneiric before pulling things all the way back... But then I am usually at the airport 2 hrs ahead of time... -Stefan
On 08/25/2011 02:36 PM, Herton Ronaldo Krzesinski wrote: > On Tue, Aug 23, 2011 at 03:08:44PM -0600, Tim Gardner wrote: >> These patches in Oneiric seem to be working pretty well. With these >> changes and sufficient server capability I've seen build times cut >> in half. Now you'll be able to control concurrency from the command >> line, e.g., >> >> dpkg-buildpackage -j`getconf _NPROCESSORS_ONLN` -B >> >> http://bugs.launchpad.net/bugs/832352 >> >> The following changes since commit 2f28aed962765c698f9ed3bc2385dbe79ada2514: >> Seth Forshee (1): >> drm/i915: Remove BUG_ON from i915_gem_evict_something >> >> are available in the git repository at: >> >> git://kernel.ubuntu.com/rtg/ubuntu-lucid.git makefile-improvenets >> >> Tim Gardner (11): >> UBUNTU: [Config] Simplify binary-udebs dependencies >> UBUNTU: [Config] kernel preparation cannot be parallelized >> UBUNTU: [Config] Linearize module/abi checks >> UBUNTU: [Config] Linearize and simplify tree preparation rules >> UBUNTU: [Config] Build kernel image in parallel with modules >> UBUNTU: [Config] Set concurrency for kmake invocations >> UBUNTU: [Config] Improve install-arch-headers speed >> UBUNTU: [Config] Fix binary-perarch dependencies >> UBUNTU: [Config] Removed stamp-flavours target >> UBUNTU: [Config] Serialize binary indep targets >> UBUNTU: [Config] Use build stamp directly > > I'm no make expert, but I noted some things. > My goals with these changes are to make the minimum number of changes whilst simplifying the rules and enabling parallel building where it makes sense. > In changes > > UBUNTU: [Config] kernel preparation cannot be parallelized > UBUNTU: [Config] Set concurrency for kmake invocations > > you force -j1. Indeed scripts target depend on oldconfig/prepare first: > > $(build_cd) $(kmake) $(build_O) -j1 silentoldconfig prepare scripts > > but each target could benefit from parallelism (which I expect to be the > case of scripts target), so a better invocation would be: > > $(build_cd) $(kmake) $(build_O) silentoldconfig > $(build_cd) $(kmake) $(build_O) prepare > $(build_cd) $(kmake) $(build_O) scripts > > Also, prepare alread implies/runs silentoldconfig, so we could simplify > even more I think: > > $(build_cd) $(kmake) $(build_O) prepare > $(build_cd) $(kmake) $(build_O) scripts > > I don't expect significant improvement, but would be better. > In the interest of not over complicating things I chose to leave this rule alone. It runs fast enough that its not worth optimizing. > In change > > UBUNTU: [Config] Linearize module/abi checks > > you could remove the install -d below "+$(abidir)/%.modules: abi-check-%" > since through abi-check-% -> $(abidir)/% we already get the directory > created. > > Also, on a quick look, seems this change is unecessary: > abi-check script doesn't seem change/care about the modules list > module-check script also only cares about modules list > Unless I'm missing something else. > > > In change > > UBUNTU: [Config] Serialize binary indep targets > > I'm in doubt, why serialize it? > While none of the indep packages are dependent, creating them linearly (since they are quite small) makes the rule set very straightforward. I don't think there is much to be gained with an independent rule set here. > > The rest looks ok to me, it's great having a better build time. > >> >> debian/rules.d/2-binary-arch.mk | 35 >> ++++++++++++++--------------------- >> debian/rules.d/3-binary-indep.mk | 29 ++++++++++++----------------- >> debian/rules.d/4-checks.mk | 10 ++-------- >> debian/rules.d/5-udebs.mk | 4 ++-- >> 4 files changed, 30 insertions(+), 48 deletions(-) >> -- >> Tim Gardner tim.gardner@canonical.com >> >> -- >> kernel-team mailing list >> kernel-team@lists.ubuntu.com >> https://lists.ubuntu.com/mailman/listinfo/kernel-team >> >
On 08/23/2011 03:08 PM, Tim Gardner wrote: > These patches in Oneiric seem to be working pretty well. With these > changes and sufficient server capability I've seen build times cut in > half. Now you'll be able to control concurrency from the command line, > e.g., > > dpkg-buildpackage -j`getconf _NPROCESSORS_ONLN` -B > > http://bugs.launchpad.net/bugs/832352 > > The following changes since commit > 2f28aed962765c698f9ed3bc2385dbe79ada2514: > Seth Forshee (1): > drm/i915: Remove BUG_ON from i915_gem_evict_something > > are available in the git repository at: > > git://kernel.ubuntu.com/rtg/ubuntu-lucid.git makefile-improvenets > > Tim Gardner (11): > UBUNTU: [Config] Simplify binary-udebs dependencies > UBUNTU: [Config] kernel preparation cannot be parallelized > UBUNTU: [Config] Linearize module/abi checks > UBUNTU: [Config] Linearize and simplify tree preparation rules > UBUNTU: [Config] Build kernel image in parallel with modules > UBUNTU: [Config] Set concurrency for kmake invocations > UBUNTU: [Config] Improve install-arch-headers speed > UBUNTU: [Config] Fix binary-perarch dependencies > UBUNTU: [Config] Removed stamp-flavours target > UBUNTU: [Config] Serialize binary indep targets > UBUNTU: [Config] Use build stamp directly > > debian/rules.d/2-binary-arch.mk | 35 ++++++++++++++--------------------- > debian/rules.d/3-binary-indep.mk | 29 ++++++++++++----------------- > debian/rules.d/4-checks.mk | 10 ++-------- > debian/rules.d/5-udebs.mk | 4 ++-- > 4 files changed, 30 insertions(+), 48 deletions(-) I've done plenty of testing, so I think its time to let this cook in master-next for a bit in the pre-proposed PPA. rtg
These patches in Oneiric seem to be working pretty well. With these changes and sufficient server capability I've seen build times cut in half. Now you'll be able to control concurrency from the command line, e.g., dpkg-buildpackage -j`getconf _NPROCESSORS_ONLN` -B http://bugs.launchpad.net/bugs/832352 The following changes since commit 2f28aed962765c698f9ed3bc2385dbe79ada2514: Seth Forshee (1): drm/i915: Remove BUG_ON from i915_gem_evict_something are available in the git repository at: git://kernel.ubuntu.com/rtg/ubuntu-lucid.git makefile-improvenets Tim Gardner (11): UBUNTU: [Config] Simplify binary-udebs dependencies UBUNTU: [Config] kernel preparation cannot be parallelized UBUNTU: [Config] Linearize module/abi checks UBUNTU: [Config] Linearize and simplify tree preparation rules UBUNTU: [Config] Build kernel image in parallel with modules UBUNTU: [Config] Set concurrency for kmake invocations UBUNTU: [Config] Improve install-arch-headers speed UBUNTU: [Config] Fix binary-perarch dependencies UBUNTU: [Config] Removed stamp-flavours target UBUNTU: [Config] Serialize binary indep targets UBUNTU: [Config] Use build stamp directly debian/rules.d/2-binary-arch.mk | 35 ++++++++++++++--------------------- debian/rules.d/3-binary-indep.mk | 29 ++++++++++++----------------- debian/rules.d/4-checks.mk | 10 ++-------- debian/rules.d/5-udebs.mk | 4 ++-- 4 files changed, 30 insertions(+), 48 deletions(-)