Message ID | 20210721221338.159114-1-apw@canonical.com |
---|---|
Headers | show |
Series | LP:1928921 LRMv5 -- switch primary version handling to kernel-versions data set | expand |
On 22.07.21 00:13, Andy Whitcroft wrote: > tl;dr move primary dkms-versions out of tree and consume those during package > update time. > > Currently the primary dkms-versions data is committed directly to the main > kernel packages. This allows this information to cascade reliably to all > derivatives and their associated LRM packages. It also ensures that the entire > derivative (and dependants) tree has the same versions even if the archive > changes in the middle of the SRU cycle. However, once the primaries are closed > it is increasingly hard to change this data. Any partial tree update > such as is needed during an LRM-only respin requires special handling which > increases risk and complexity. > > We could update the versions from the commited archive versions in every > package, but this exposes us to risk of version skew between both derivatives > and even within a dependant tree; this is particularly serious with Nvidia > where only one version is valid in each nvidia series. > > To address this we propose moving the dkms-versions data into its own > repository[1]. This repository is then updated at the start of each cycle > (more often if necessary) primarily as a cycle opening task. The current cycle > data is then pulled into the packages as they are ready to close ensuring the > builds are still independant of the data-set. > > It should be noted that the dkms-versions information is not static. It > commonly differs between SRU cycles. When spinning (or respinning) a kernel > you need to use the version set for that cycle, as updated during and for that > cycle. This is particularly important when respinning the previous cycle when > the current cycle has already started with updated versions. > > To this end the kernel-versions repository contains branches for each cycle > (though typically these will be points in time of a linear progression of the > version information). This allows us to select the appropriate point in the > history to extract those versions, while still allowing each of them to evolve > as needed independantly. This model is extended to include security spins > (sYYYY.MM.DD form) which are selected from a private repository in the security > team. Updates to dkms-versions information is primarily performed by the > `update-dkms-versions` script within the repository which understands the > vaguaries of Nvidia versioning and the preferred pockets to scan. > > In order to make selection of the appropriate spin to further select the right > point in the kernel-versions history we desire the local tree to contain the > associated cycle information. We have recently started including the spin > number in the debian*/tracking-bug file in the main kernel package. By simply > syncing this file into the other trees as needed, we can ensure this data is > accurate in the majority case. For security spins we will need to override it > manually. > > Following this email is a patch stack to update our LRM packages to LRMv5 which > includes the new dkms-versions handling. It consists of four patches which make > up the proposed change. I include them here to aid review: > > UBUNTU: [Packaging] update-version -- allow specification of the master version > UBUNTU: [Packaging] update-version -- call out to update-dkms-versions > UBUNTU: [Packaging] update-version -- copy over tracking-bug information > UBUNTU: [Packaging] update-dkms-versions -- update from kernel-versions > > Rather than applying these manually to all of our 41 LRM packages I propose to > apply this change via an automated update squashing these into an LRMv5 update; this > mirrors the process we used for LRMv3 and LRMv4. > > Proposing for all of our LRM packages. > > Separatly we should include the same `update-dkms-versions` script in our main > kernels as they also include the dkms-versions data; though they only consume > the zfs and wireguard elements. This replaces the existing > `update-versions-dkms` script as we wish to change the frequency of use of this > script and this change helps catch tooling which has not adapted. I would > propose applying that under the same bug. > > Finally we would need to modify our processes to always run > `update-dkms-versions` when present after `link-to-tracker` has run. > > Comments. > > -apw > > [1] https://code.launchpad.net/~canonical-kernel/+git/kernel-versions > > Branch: autogen5 > > Andy Whitcroft (4): > UBUNTU: [Packaging] update-dkms-versions -- update from kernel-versions > UBUNTU: [Packaging] update-version -- copy over tracking-bug information > UBUNTU: [Packaging] update-version -- call out to update-dkms-versions > UBUNTU: [Packaging] update-version -- allow specification of the master version > > update-dkms-versions | 150 +++++++++++++++++++++++++++++++++++++++++++ > update-version | 43 +++++++++---- > 2 files changed, 181 insertions(+), 12 deletions(-) > create mode 100755 update-dkms-versions > Maybe it also would be helpful to describe the layout of kernel-versions + |\ | +- kernel-versions | \ | +- <series> (file containing dkms version data) | +- <series> |\ | +- map | \ | +- kernel-versions | \ | +- <kernel version> (file containing which <series> to use | +- <kernel version> | +- update-dkms-versions Overall this has been used for a couple of cases, so should be ok to move to production. Acked-by: Stefan Bader <stefan.bader@canonical.com>
On 22.07.21 00:13, Andy Whitcroft wrote: > tl;dr move primary dkms-versions out of tree and consume those during package > update time. > > Currently the primary dkms-versions data is committed directly to the main > kernel packages. This allows this information to cascade reliably to all > derivatives and their associated LRM packages. It also ensures that the entire > derivative (and dependants) tree has the same versions even if the archive > changes in the middle of the SRU cycle. However, once the primaries are closed > it is increasingly hard to change this data. Any partial tree update > such as is needed during an LRM-only respin requires special handling which > increases risk and complexity. > > We could update the versions from the commited archive versions in every > package, but this exposes us to risk of version skew between both derivatives > and even within a dependant tree; this is particularly serious with Nvidia > where only one version is valid in each nvidia series. > > To address this we propose moving the dkms-versions data into its own > repository[1]. This repository is then updated at the start of each cycle > (more often if necessary) primarily as a cycle opening task. The current cycle > data is then pulled into the packages as they are ready to close ensuring the > builds are still independant of the data-set. > > It should be noted that the dkms-versions information is not static. It > commonly differs between SRU cycles. When spinning (or respinning) a kernel > you need to use the version set for that cycle, as updated during and for that > cycle. This is particularly important when respinning the previous cycle when > the current cycle has already started with updated versions. > > To this end the kernel-versions repository contains branches for each cycle > (though typically these will be points in time of a linear progression of the > version information). This allows us to select the appropriate point in the > history to extract those versions, while still allowing each of them to evolve > as needed independantly. This model is extended to include security spins > (sYYYY.MM.DD form) which are selected from a private repository in the security > team. Updates to dkms-versions information is primarily performed by the > `update-dkms-versions` script within the repository which understands the > vaguaries of Nvidia versioning and the preferred pockets to scan. > > In order to make selection of the appropriate spin to further select the right > point in the kernel-versions history we desire the local tree to contain the > associated cycle information. We have recently started including the spin > number in the debian*/tracking-bug file in the main kernel package. By simply > syncing this file into the other trees as needed, we can ensure this data is > accurate in the majority case. For security spins we will need to override it > manually. > > Following this email is a patch stack to update our LRM packages to LRMv5 which > includes the new dkms-versions handling. It consists of four patches which make > up the proposed change. I include them here to aid review: > > UBUNTU: [Packaging] update-version -- allow specification of the master version > UBUNTU: [Packaging] update-version -- call out to update-dkms-versions > UBUNTU: [Packaging] update-version -- copy over tracking-bug information > UBUNTU: [Packaging] update-dkms-versions -- update from kernel-versions > > Rather than applying these manually to all of our 41 LRM packages I propose to > apply this change via an automated update squashing these into an LRMv5 update; this > mirrors the process we used for LRMv3 and LRMv4. > > Proposing for all of our LRM packages. > > Separatly we should include the same `update-dkms-versions` script in our main > kernels as they also include the dkms-versions data; though they only consume > the zfs and wireguard elements. This replaces the existing > `update-versions-dkms` script as we wish to change the frequency of use of this > script and this change helps catch tooling which has not adapted. I would > propose applying that under the same bug. > > Finally we would need to modify our processes to always run > `update-dkms-versions` when present after `link-to-tracker` has run. > > Comments. > > -apw > > [1] https://code.launchpad.net/~canonical-kernel/+git/kernel-versions > > Branch: autogen5 > > Andy Whitcroft (4): > UBUNTU: [Packaging] update-dkms-versions -- update from kernel-versions > UBUNTU: [Packaging] update-version -- copy over tracking-bug information > UBUNTU: [Packaging] update-version -- call out to update-dkms-versions > UBUNTU: [Packaging] update-version -- allow specification of the master version > > update-dkms-versions | 150 +++++++++++++++++++++++++++++++++++++++++++ > update-version | 43 +++++++++---- > 2 files changed, 181 insertions(+), 12 deletions(-) > create mode 100755 update-dkms-versions > Acked-by: Kleber Sacilotto de Souza <kleber.souza@canonical.com> Thanks