| Message ID | 20210308150004.1746089-1-apw@canonical.com |
|---|---|
| Headers | show |
| Series | LP#1918134 -- LRMv4 switch to signing with Ubuntu Kernel Modules signing key | expand |
On Mon, Mar 08, 2021 at 02:59:55PM +0000, Andy Whitcroft wrote: > Currently we build Nvidia packages in the main kernel package (linux, > linux-aws etc). We do this to sign the resulting .ko's into the kernel > ephemeral modules signing key. We package up only the added signature > data (as .sig files) and destroy the remainder. We later rebuild the .o's > (and temporarily the .ko's) in linux-restricted-modules* and include the > .o's so produced and the .sig's as produced by the main kernel build in > LRM's binary packages. These are then reconstructed on installation on > the end-user system. > > Now this has several unfortunate properties. Firstly we have to access > and build dkms packages from the restricted component in the main kernel > package build (which is hard and error prone). Secondly, we must be able > to reproducibly build the .ko's as we build them in both the main package > and LRM. (Recently a toolchain change seems to have made this build no > longer reproducible leading to modules which do not match the produced > signatures and are thus not loadable.) Finally, in order to bump the > version of any particular Nvidia stream (or add or remove a stream) we > have to rebuild the main kernel package in order to get signatures which > is very costly and leads to unnecessary kernel updates for people who do > not even use Nvidia modules. > > As background, in recent kernel uploads we have included two new static > keys into the kernel `.builtin_trusted_keys`; the Canonical Livepatch key, > and the Ubuntu Kernel Modules key. This second key is designed to allow > signing of driver .ko's into the kernel even after the build is completed. > With this in place it is no longer necessary for us to build the Nvidia > packages in the main kernel package to obtain signatures. > > Signing of .ko's with the Ubuntu Drivers Key (UDK) is performed via a > signing custom binary upload containing those .ko's. This signing either > produces a signed .ko file, or optionally a detached .sig signature. > For our purposes the latter is our desired form as we cannot distribute > the combined .ko files. It is very important to note that as we > cannot distribute the combined .ko files for these Nvidia packages > (due to liscencing constraints) that we also cannot allow the resultant > custom binary upload to be distributed either; ie it cannot end up on the > launchpad librarian or published into the public archive. This constraint > is a key design feature if the implementation described below. > > In the new world order (LRMv4) the linux-restricted-modules (LRM) package > is responsible for the only build of the nvidia .o's and the helpers which > allow those to be combined on the end-user system to produce final loadable > .ko's; this avoids the build reproducibility issues. Two new packages are > introduced, linux-restricted-generate (LRG) and linux-restricted-signatures > (LRS). LRG consumes the raw .o's and helpers and uses those to reconstruct > the unsigned .ko's and places those into the signing custom binary upload. > LRS simply downloads the resulting signatures from the archive and > expresses those directly into the filesystem in the same directory as the > .o's expressed by LRM. LRM's binary packages then directly depend on these > signatures such that when the postinst there runs we are able to add those > signatures to the unsigned .ko's forming loadable .ko's for the kernel. > > In order to allow LRG to simply consume the raw .o's and build helpers via > a Build-Depend: we have restructured LRM's binary packages. We split the > raw .o's and build helpers from the build dependancies and postinst/prerm > support. The former is needed both by LRG and on the end-user system, > the latter is only needed on the end-user system to construct and > destroy the signed .ko's. The raw .o's and build scripts are moved > into new linux-objects-nvidia-460-ABI-FLAVOUR packages. The existing > linux-modules-nvidia-460-ABI-FLAVOUR packages simply depend on that and > the associated signature package, and carry the postinst/prerm support. > > Building this package set is complex as the LRG package cannot be built > in a public PPA as that would cause the signing custom binary upload to > be redistributed by the launchpad librarian. We therefore must build > LRG and LRS in a private PPA. The built LRG and LRS must then be signed > using the Ubuntu Kernel Modules key (again privatly) and the resultant > LRS returned for distribution; this occurs via a private signing PPA > containing the appropriate key; LRG is abandoned at this stage and takes > no futher part. As LRM and LRS are redistributable (and indeed both are > required to install the drivers) LRS is returned to the public build PPA > where LRM was built and enters the normal pipeline. > > As LRG depends on binary packages produced by LRM the private PPA in which > LRG will build must depend on the PPA in which LRM builds. To support > this we introduce an adjunct PPA for each public build PPA which will > produce kernels which contain an LRM component. These are needed for > both of the stable PPAs (ppa and ppa2) and the development PPAs (bootstrap > and unstable); each is simply named using the primary PPA name with a -ps > (Private Signing) suffix. Below is a flow-chart showing the lifecycles > for these packages: > > LRM LRS LRG > | .LRS(-b). | | > v v | v v > +------------+ | +------------+ > | ppa | | | ppa-ps | > +------------+ | +------------+ > | | | | > | | LRS LRG(-b) > v | v v > +------------+ | +------------+ > | -proposed | `---| signing | > +------------+ +------------+ > > This new proceedure is complex and error prone as (nominally) this requires > the addition and maintenance of two new packages. Also of the three > packages in this set one is initially uploaded to the public PPA with the > rest of the kernel (linux, linux-signed, and linux-meta) and the other two > must be uploaded to a specific related private PPA (it must depend on the > associated public PPA); indeed if LRG is uploaded to the public PPA we may > end up redistributing nvidia .ko's which must be prevented at all costs. > > To avoid the complexity and associated risk of misplacing these new > packages the LRG and LRS package source are folded into the LRM package. > This package will continue to be maintained in the normal way and uploaded > to the public build PPA as normal, meaning no process change for crankers. > LRS and LRG are (intentionally) generable from LRM simply by changing the > name of the source package in debian/changelog. A cron based helper will > monitor the primary PPAs for new LRM uploads and automatically convert > that into the necessary LRG/LRS pairing and upload it to the correct > adjunct PPA. The tracking bugs associated with our uploads will drive > the process from there to sign and return LRS to the primary build PPA. > > The patch stack below represents the nominal conversion of the > hirsute:linux package from LRMv3 to LRMv4. However (as we did with LRMv3) > I proposed to apply this update programatically to all LRM packages > once this is reviewed. > > This stack does not contain the (further) changes we will apply to the main > kernel packages to stop building the nvidia signatures there. We also > continue to assume that the primary kernel package for a version-series > (bionic:linux) still carries the definative debian/dkms-versions file so > that we retain consistent versioning on these packages[1]; though this > does not preclude LRMs being respun independently during a cycle. > > -apw > > [1] the next phase of this work will revisit the home for this version > data going forward, and the aquisition of the DKMS packages during > the builds, and later. I think I've got a pretty good handle now on of how this all works, and it makes sense to me. The process is convoluted, with sort of a circular chain of dependencies, but that's unfortunately necessary and it brings maintenance benefits that I think are worthwhile. Having looked at a tree and a test build with these changes, the results look reasonable. Acked-by: Seth Forshee <seth.forshee@canonical.com>
On 08.03.21 15:59, Andy Whitcroft wrote: > Currently we build Nvidia packages in the main kernel package (linux, > linux-aws etc). We do this to sign the resulting .ko's into the kernel > ephemeral modules signing key. We package up only the added signature > data (as .sig files) and destroy the remainder. We later rebuild the .o's > (and temporarily the .ko's) in linux-restricted-modules* and include the > .o's so produced and the .sig's as produced by the main kernel build in > LRM's binary packages. These are then reconstructed on installation on > the end-user system. > > Now this has several unfortunate properties. Firstly we have to access > and build dkms packages from the restricted component in the main kernel > package build (which is hard and error prone). Secondly, we must be able > to reproducibly build the .ko's as we build them in both the main package > and LRM. (Recently a toolchain change seems to have made this build no > longer reproducible leading to modules which do not match the produced > signatures and are thus not loadable.) Finally, in order to bump the > version of any particular Nvidia stream (or add or remove a stream) we > have to rebuild the main kernel package in order to get signatures which > is very costly and leads to unnecessary kernel updates for people who do > not even use Nvidia modules. > > As background, in recent kernel uploads we have included two new static > keys into the kernel `.builtin_trusted_keys`; the Canonical Livepatch key, > and the Ubuntu Kernel Modules key. This second key is designed to allow > signing of driver .ko's into the kernel even after the build is completed. > With this in place it is no longer necessary for us to build the Nvidia > packages in the main kernel package to obtain signatures. > > Signing of .ko's with the Ubuntu Drivers Key (UDK) is performed via a > signing custom binary upload containing those .ko's. This signing either > produces a signed .ko file, or optionally a detached .sig signature. > For our purposes the latter is our desired form as we cannot distribute > the combined .ko files. It is very important to note that as we > cannot distribute the combined .ko files for these Nvidia packages > (due to liscencing constraints) that we also cannot allow the resultant > custom binary upload to be distributed either; ie it cannot end up on the > launchpad librarian or published into the public archive. This constraint > is a key design feature if the implementation described below. > > In the new world order (LRMv4) the linux-restricted-modules (LRM) package > is responsible for the only build of the nvidia .o's and the helpers which > allow those to be combined on the end-user system to produce final loadable > .ko's; this avoids the build reproducibility issues. Two new packages are > introduced, linux-restricted-generate (LRG) and linux-restricted-signatures > (LRS). LRG consumes the raw .o's and helpers and uses those to reconstruct > the unsigned .ko's and places those into the signing custom binary upload. > LRS simply downloads the resulting signatures from the archive and > expresses those directly into the filesystem in the same directory as the > .o's expressed by LRM. LRM's binary packages then directly depend on these > signatures such that when the postinst there runs we are able to add those > signatures to the unsigned .ko's forming loadable .ko's for the kernel. > > In order to allow LRG to simply consume the raw .o's and build helpers via > a Build-Depend: we have restructured LRM's binary packages. We split the > raw .o's and build helpers from the build dependancies and postinst/prerm > support. The former is needed both by LRG and on the end-user system, > the latter is only needed on the end-user system to construct and > destroy the signed .ko's. The raw .o's and build scripts are moved > into new linux-objects-nvidia-460-ABI-FLAVOUR packages. The existing > linux-modules-nvidia-460-ABI-FLAVOUR packages simply depend on that and > the associated signature package, and carry the postinst/prerm support. > > Building this package set is complex as the LRG package cannot be built > in a public PPA as that would cause the signing custom binary upload to > be redistributed by the launchpad librarian. We therefore must build > LRG and LRS in a private PPA. The built LRG and LRS must then be signed > using the Ubuntu Kernel Modules key (again privatly) and the resultant > LRS returned for distribution; this occurs via a private signing PPA > containing the appropriate key; LRG is abandoned at this stage and takes > no futher part. As LRM and LRS are redistributable (and indeed both are > required to install the drivers) LRS is returned to the public build PPA > where LRM was built and enters the normal pipeline. > > As LRG depends on binary packages produced by LRM the private PPA in which > LRG will build must depend on the PPA in which LRM builds. To support > this we introduce an adjunct PPA for each public build PPA which will > produce kernels which contain an LRM component. These are needed for > both of the stable PPAs (ppa and ppa2) and the development PPAs (bootstrap > and unstable); each is simply named using the primary PPA name with a -ps > (Private Signing) suffix. Below is a flow-chart showing the lifecycles > for these packages: > > LRM LRS LRG > | .LRS(-b). | | > v v | v v > +------------+ | +------------+ > | ppa | | | ppa-ps | > +------------+ | +------------+ > | | | | > | | LRS LRG(-b) > v | v v > +------------+ | +------------+ > | -proposed | `---| signing | > +------------+ +------------+ > > This new proceedure is complex and error prone as (nominally) this requires > the addition and maintenance of two new packages. Also of the three > packages in this set one is initially uploaded to the public PPA with the > rest of the kernel (linux, linux-signed, and linux-meta) and the other two > must be uploaded to a specific related private PPA (it must depend on the > associated public PPA); indeed if LRG is uploaded to the public PPA we may > end up redistributing nvidia .ko's which must be prevented at all costs. > > To avoid the complexity and associated risk of misplacing these new > packages the LRG and LRS package source are folded into the LRM package. > This package will continue to be maintained in the normal way and uploaded > to the public build PPA as normal, meaning no process change for crankers. > LRS and LRG are (intentionally) generable from LRM simply by changing the > name of the source package in debian/changelog. A cron based helper will > monitor the primary PPAs for new LRM uploads and automatically convert > that into the necessary LRG/LRS pairing and upload it to the correct > adjunct PPA. The tracking bugs associated with our uploads will drive > the process from there to sign and return LRS to the primary build PPA. > > The patch stack below represents the nominal conversion of the > hirsute:linux package from LRMv3 to LRMv4. However (as we did with LRMv3) > I proposed to apply this update programatically to all LRM packages > once this is reviewed. > > This stack does not contain the (further) changes we will apply to the main > kernel packages to stop building the nvidia signatures there. We also > continue to assume that the primary kernel package for a version-series > (bionic:linux) still carries the definative debian/dkms-versions file so > that we retain consistent versioning on these packages[1]; though this > does not preclude LRMs being respun independently during a cycle. > > -apw > > [1] the next phase of this work will revisit the home for this version > data going forward, and the aquisition of the DKMS packages during > the builds, and later. > --- > > Branch: autogen4 > > Andy Whitcroft (9): > UBUNTU: [Packaging] parameterise package generator based on package > UBUNTU: [Packaging] gen-rules -- clean up debian/rules fragment > UBUNTU: [Packaging] dkms-build--nvidia-N -- handle symbol versioning being enabled > UBUNTU: [Packaging] dkms-build--nvidia-N -- follow changes to lds naming > UBUNTU: [Packaging] dkms-build--nvidia-N -- add unsigned build mode to helpers > UBUNTU: [Packaging] linux-restricted-generate -- generate unsigned modules for signing > UBUNTU: [Packaging] linux-restricted-signatures -- publish clean signatures > UBUNTU: [Packaging] linux-restricted-modules -- consume published signatures > UBUNTU: [Packaging] update-version -- handle forward-backport versioning > > debian/rules.in | 26 +--- > debian/rules.lrg | 31 ++++ > debian/rules.lrs | 12 ++ > debian/scripts/dkms-build--nvidia-N | 63 ++++---- > debian/scripts/gen-rules | 179 +--------------------- > debian/scripts/gen-rules.lrg | 138 +++++++++++++++++ > debian/scripts/gen-rules.lrm | 226 ++++++++++++++++++++++++++++ > debian/scripts/gen-rules.lrs | 100 ++++++++++++ > debian/source/options | 2 +- > download-signed | 183 ++++++++++++++++++++++ > scripts/module-common.lds | 26 ---- > update-version | 5 +- > 12 files changed, 741 insertions(+), 250 deletions(-) > create mode 100755 debian/rules.lrg > create mode 100755 debian/rules.lrs > create mode 100755 debian/scripts/gen-rules.lrg > create mode 100755 debian/scripts/gen-rules.lrm > create mode 100755 debian/scripts/gen-rules.lrs > create mode 100755 download-signed > delete mode 100644 scripts/module-common.lds > Some details I ignored as those should either work and be tested or not. When we follow the approach of first see this in action on devel and then roll this out programmatically, then we should be on the safe side. Acked-by: Stefan Bader <stefan.bader@canonical.com>
Currently we build Nvidia packages in the main kernel package (linux, linux-aws etc). We do this to sign the resulting .ko's into the kernel ephemeral modules signing key. We package up only the added signature data (as .sig files) and destroy the remainder. We later rebuild the .o's (and temporarily the .ko's) in linux-restricted-modules* and include the .o's so produced and the .sig's as produced by the main kernel build in LRM's binary packages. These are then reconstructed on installation on the end-user system. Now this has several unfortunate properties. Firstly we have to access and build dkms packages from the restricted component in the main kernel package build (which is hard and error prone). Secondly, we must be able to reproducibly build the .ko's as we build them in both the main package and LRM. (Recently a toolchain change seems to have made this build no longer reproducible leading to modules which do not match the produced signatures and are thus not loadable.) Finally, in order to bump the version of any particular Nvidia stream (or add or remove a stream) we have to rebuild the main kernel package in order to get signatures which is very costly and leads to unnecessary kernel updates for people who do not even use Nvidia modules. As background, in recent kernel uploads we have included two new static keys into the kernel `.builtin_trusted_keys`; the Canonical Livepatch key, and the Ubuntu Kernel Modules key. This second key is designed to allow signing of driver .ko's into the kernel even after the build is completed. With this in place it is no longer necessary for us to build the Nvidia packages in the main kernel package to obtain signatures. Signing of .ko's with the Ubuntu Drivers Key (UDK) is performed via a signing custom binary upload containing those .ko's. This signing either produces a signed .ko file, or optionally a detached .sig signature. For our purposes the latter is our desired form as we cannot distribute the combined .ko files. It is very important to note that as we cannot distribute the combined .ko files for these Nvidia packages (due to liscencing constraints) that we also cannot allow the resultant custom binary upload to be distributed either; ie it cannot end up on the launchpad librarian or published into the public archive. This constraint is a key design feature if the implementation described below. In the new world order (LRMv4) the linux-restricted-modules (LRM) package is responsible for the only build of the nvidia .o's and the helpers which allow those to be combined on the end-user system to produce final loadable .ko's; this avoids the build reproducibility issues. Two new packages are introduced, linux-restricted-generate (LRG) and linux-restricted-signatures (LRS). LRG consumes the raw .o's and helpers and uses those to reconstruct the unsigned .ko's and places those into the signing custom binary upload. LRS simply downloads the resulting signatures from the archive and expresses those directly into the filesystem in the same directory as the .o's expressed by LRM. LRM's binary packages then directly depend on these signatures such that when the postinst there runs we are able to add those signatures to the unsigned .ko's forming loadable .ko's for the kernel. In order to allow LRG to simply consume the raw .o's and build helpers via a Build-Depend: we have restructured LRM's binary packages. We split the raw .o's and build helpers from the build dependancies and postinst/prerm support. The former is needed both by LRG and on the end-user system, the latter is only needed on the end-user system to construct and destroy the signed .ko's. The raw .o's and build scripts are moved into new linux-objects-nvidia-460-ABI-FLAVOUR packages. The existing linux-modules-nvidia-460-ABI-FLAVOUR packages simply depend on that and the associated signature package, and carry the postinst/prerm support. Building this package set is complex as the LRG package cannot be built in a public PPA as that would cause the signing custom binary upload to be redistributed by the launchpad librarian. We therefore must build LRG and LRS in a private PPA. The built LRG and LRS must then be signed using the Ubuntu Kernel Modules key (again privatly) and the resultant LRS returned for distribution; this occurs via a private signing PPA containing the appropriate key; LRG is abandoned at this stage and takes no futher part. As LRM and LRS are redistributable (and indeed both are required to install the drivers) LRS is returned to the public build PPA where LRM was built and enters the normal pipeline. As LRG depends on binary packages produced by LRM the private PPA in which LRG will build must depend on the PPA in which LRM builds. To support this we introduce an adjunct PPA for each public build PPA which will produce kernels which contain an LRM component. These are needed for both of the stable PPAs (ppa and ppa2) and the development PPAs (bootstrap and unstable); each is simply named using the primary PPA name with a -ps (Private Signing) suffix. Below is a flow-chart showing the lifecycles for these packages: LRM LRS LRG | .LRS(-b). | | v v | v v +------------+ | +------------+ | ppa | | | ppa-ps | +------------+ | +------------+ | | | | | | LRS LRG(-b) v | v v +------------+ | +------------+ | -proposed | `---| signing | +------------+ +------------+ This new proceedure is complex and error prone as (nominally) this requires the addition and maintenance of two new packages. Also of the three packages in this set one is initially uploaded to the public PPA with the rest of the kernel (linux, linux-signed, and linux-meta) and the other two must be uploaded to a specific related private PPA (it must depend on the associated public PPA); indeed if LRG is uploaded to the public PPA we may end up redistributing nvidia .ko's which must be prevented at all costs. To avoid the complexity and associated risk of misplacing these new packages the LRG and LRS package source are folded into the LRM package. This package will continue to be maintained in the normal way and uploaded to the public build PPA as normal, meaning no process change for crankers. LRS and LRG are (intentionally) generable from LRM simply by changing the name of the source package in debian/changelog. A cron based helper will monitor the primary PPAs for new LRM uploads and automatically convert that into the necessary LRG/LRS pairing and upload it to the correct adjunct PPA. The tracking bugs associated with our uploads will drive the process from there to sign and return LRS to the primary build PPA. The patch stack below represents the nominal conversion of the hirsute:linux package from LRMv3 to LRMv4. However (as we did with LRMv3) I proposed to apply this update programatically to all LRM packages once this is reviewed. This stack does not contain the (further) changes we will apply to the main kernel packages to stop building the nvidia signatures there. We also continue to assume that the primary kernel package for a version-series (bionic:linux) still carries the definative debian/dkms-versions file so that we retain consistent versioning on these packages[1]; though this does not preclude LRMs being respun independently during a cycle. -apw [1] the next phase of this work will revisit the home for this version data going forward, and the aquisition of the DKMS packages during the builds, and later. --- Branch: autogen4 Andy Whitcroft (9): UBUNTU: [Packaging] parameterise package generator based on package UBUNTU: [Packaging] gen-rules -- clean up debian/rules fragment UBUNTU: [Packaging] dkms-build--nvidia-N -- handle symbol versioning being enabled UBUNTU: [Packaging] dkms-build--nvidia-N -- follow changes to lds naming UBUNTU: [Packaging] dkms-build--nvidia-N -- add unsigned build mode to helpers UBUNTU: [Packaging] linux-restricted-generate -- generate unsigned modules for signing UBUNTU: [Packaging] linux-restricted-signatures -- publish clean signatures UBUNTU: [Packaging] linux-restricted-modules -- consume published signatures UBUNTU: [Packaging] update-version -- handle forward-backport versioning debian/rules.in | 26 +--- debian/rules.lrg | 31 ++++ debian/rules.lrs | 12 ++ debian/scripts/dkms-build--nvidia-N | 63 ++++---- debian/scripts/gen-rules | 179 +--------------------- debian/scripts/gen-rules.lrg | 138 +++++++++++++++++ debian/scripts/gen-rules.lrm | 226 ++++++++++++++++++++++++++++ debian/scripts/gen-rules.lrs | 100 ++++++++++++ debian/source/options | 2 +- download-signed | 183 ++++++++++++++++++++++ scripts/module-common.lds | 26 ---- update-version | 5 +- 12 files changed, 741 insertions(+), 250 deletions(-) create mode 100755 debian/rules.lrg create mode 100755 debian/rules.lrs create mode 100755 debian/scripts/gen-rules.lrg create mode 100755 debian/scripts/gen-rules.lrm create mode 100755 debian/scripts/gen-rules.lrs create mode 100755 download-signed delete mode 100644 scripts/module-common.lds