| Message ID | 20230727232235.449788-1-christian@aperture.us |
|---|---|
| State | Accepted |
| Headers | show |
| Series | [v1,1/1] package/containerd: bump version to v1.6.22 | expand |
On Thu, 27 Jul 2023 16:22:35 -0700 Christian Stewart via buildroot <buildroot@buildroot.org> wrote: > Bugfixes and updates. > > https://github.com/containerd/containerd/releases/tag/v1.6.22 > > Signed-off-by: Christian Stewart <christian@aperture.us> > --- > package/containerd/containerd.hash | 2 +- > package/containerd/containerd.mk | 2 +- > 2 files changed, 2 insertions(+), 2 deletions(-) Applied to master, thanks. Thomas
>>>>> "Thomas" == Thomas Petazzoni via buildroot <buildroot@buildroot.org> writes: > On Thu, 27 Jul 2023 16:22:35 -0700 > Christian Stewart via buildroot <buildroot@buildroot.org> wrote: >> Bugfixes and updates. >> >> https://github.com/containerd/containerd/releases/tag/v1.6.22 >> >> Signed-off-by: Christian Stewart <christian@aperture.us> >> --- >> package/containerd/containerd.hash | 2 +- >> package/containerd/containerd.mk | 2 +- >> 2 files changed, 2 insertions(+), 2 deletions(-) > Applied to master, thanks. Committed to 2023.02.x and 2023.05.x, thanks.
Hi Christian, It seems that this version hangs at startup, at least when built with Buildroot 2023.02.4. Typically my systemd services just timeout. When I manually start containerd it just doesn't reach the message "containerd successfully booted in", which older version did. I've tested v1.6.23 as well, it shows the same behavior. For now I reverted to v1.6.21. Have you seen this behavior? Btw, on master, where we use Docker 24.0.x, shouldn't containerd be upgraded to the 1.7.x series? -- Stefan On 2023-07-28 01:22, Christian Stewart via buildroot wrote: > Bugfixes and updates. > > https://github.com/containerd/containerd/releases/tag/v1.6.22 > > Signed-off-by: Christian Stewart <christian@aperture.us> > --- > package/containerd/containerd.hash | 2 +- > package/containerd/containerd.mk | 2 +- > 2 files changed, 2 insertions(+), 2 deletions(-) > > diff --git a/package/containerd/containerd.hash > b/package/containerd/containerd.hash > index 345a7b16ff..8404675e30 100644 > --- a/package/containerd/containerd.hash > +++ b/package/containerd/containerd.hash > @@ -1,3 +1,3 @@ > # Computed locally > -sha256 > 9452e95455d03a00d78ae0587595d0c18555bae7912068269efa25a724efe713 > containerd-1.6.21.tar.gz > +sha256 > b109aceacc814d7a637ed94ba5ade829cd2642841d03e06971ef124fa3b86899 > containerd-1.6.22.tar.gz > sha256 > 4bbe3b885e8cd1907ab4cf9a41e862e74e24b5422297a4f2fe524e6a30ada2b4 > LICENSE > diff --git a/package/containerd/containerd.mk b/package/containerd/containerd.mk > index 83489a3abf..c4a6d854c3 100644 > --- a/package/containerd/containerd.mk > +++ b/package/containerd/containerd.mk > @@ -4,7 +4,7 @@ > # > > ################################################################################ > > -CONTAINERD_VERSION = 1.6.21 > +CONTAINERD_VERSION = 1.6.22 > CONTAINERD_SITE = $(call github,containerd,containerd,v$(CONTAINERD_VERSION)) > CONTAINERD_LICENSE = Apache-2.0 > CONTAINERD_LICENSE_FILES = LICENSE
Hi Stefan, On Thu, Sep 7, 2023 at 4:23 PM Stefan Agner <stefan@agner.ch> wrote: > > Hi Christian, > > It seems that this version hangs at startup, at least when built with > Buildroot 2023.02.4. I am running 1.7.x on my systems so I can't confirm this currently. > Btw, on master, where we use Docker 24.0.x, shouldn't containerd be > upgraded to the 1.7.x series? According to this 1.6.x is LTS and still getting updates: https://containerd.io/releases/ According to these release notes, 1.6.x "has also become a long term stable release for users who prefer releases with mostly stability improvements and wish to wait a few releases for new functionality." See: https://github.com/containerd/containerd/releases/tag/v1.7.0 Since Docker is supposed to be agnostic to different containerd versions it made sense to use the LTS. However, I am not against updating it to 1.7.x as I'm currently using & testing that branch anyway. Best, Christian Stewart
On 8 September 2023 01:28:36 CEST, Christian Stewart <christian@aperture.us> wrote: >Hi Stefan, > >On Thu, Sep 7, 2023 at 4:23 PM Stefan Agner <stefan@agner.ch> wrote: >> >> Hi Christian, >> >> It seems that this version hangs at startup, at least when built with >> Buildroot 2023.02.4. > >I am running 1.7.x on my systems so I can't confirm this currently. I've just tested 1.7.3 as well, it shows the same behavior here. I'll need to do more testing. Ideally I like to see a stack trace (via signal or debugger), but not sure how to archive this with Buildroot and golang. This is on aarch64 platforms. > >> Btw, on master, where we use Docker 24.0.x, shouldn't containerd be >> upgraded to the 1.7.x series? > >According to this 1.6.x is LTS and still getting updates: > >https://containerd.io/releases/ > >According to these release notes, 1.6.x "has also become a long term >stable release for users who prefer releases with mostly stability >improvements and wish to wait a few releases for new functionality." > >See: https://github.com/containerd/containerd/releases/tag/v1.7.0 > >Since Docker is supposed to be agnostic to different containerd >versions it made sense to use the LTS. > >However, I am not against updating it to 1.7.x as I'm currently using >& testing that branch anyway. Ok I see. Either way works for me. Best regards, Stefan > >Best, >Christian Stewart
Hi Stefan, On Thu, Sep 7, 2023, 4:51 PM Stefan Agner <stefan@agner.ch> wrote: > > > On 8 September 2023 01:28:36 CEST, Christian Stewart < > christian@aperture.us> wrote: > >Hi Stefan, > > > >On Thu, Sep 7, 2023 at 4:23 PM Stefan Agner <stefan@agner.ch> wrote: > >> > >> Hi Christian, > >> > >> It seems that this version hangs at startup, at least when built with > >> Buildroot 2023.02.4. > > > >I am running 1.7.x on my systems so I can't confirm this currently. > > I've just tested 1.7.3 as well, it shows the same behavior here. I'll need > to do more testing. Ideally I like to see a stack trace (via signal or > debugger), but not sure how to archive this with Buildroot and golang. > > This is on aarch64 platforms. > I hate to say "works on my machine," but with both versions on aarch64 it works for me. Maybe the filesystem you are using is different? I'm using ext4. Or is the kernel config not set up quite right? Maybe old kernel? Best, Christian
On 2023-09-08 04:03, Christian Stewart wrote: > Hi Stefan, > > On Thu, Sep 7, 2023, 4:51 PM Stefan Agner <stefan@agner.ch> wrote: > >> On 8 September 2023 01:28:36 CEST, Christian Stewart <christian@aperture.us> wrote: >>> Hi Stefan, >>> >>> On Thu, Sep 7, 2023 at 4:23 PM Stefan Agner <stefan@agner.ch> wrote: >>>> >>>> Hi Christian, >>>> >>>> It seems that this version hangs at startup, at least when built with >>>> Buildroot 2023.02.4. >>> >>> I am running 1.7.x on my systems so I can't confirm this currently. >> >> I've just tested 1.7.3 as well, it shows the same behavior here. I'll need to do more testing. Ideally I like to see a stack trace (via signal or debugger), but not sure how to archive this with Buildroot and golang. >> >> This is on aarch64 platforms. > > I hate to say "works on my machine," but with both versions on aarch64 it works for me. > > Maybe the filesystem you are using is different? I'm using ext4. Or is the kernel config not set up quite right? Maybe old kernel? The root file system is a squashfs, but the data is stored on ext4. We are using the latest vanilla Linux kernel 6.1.52. It seems to hang here: WARN[2023-09-08T06:17:38.515218691Z] waiting for response from boltdb open plugin=bolt From what I can tell it's some IPC mechanism or similar which is hanging. I can also reproduce it on x86-64. I guess it must be unique to our operating system configuration in some way... -- Stefan > > Best, > Christian
Hi Stefan, Did you find a solution to this? Just following up. On Thu, Sep 7, 2023 at 11:21 PM Stefan Agner <stefan@agner.ch> wrote: > > I hate to say "works on my machine," but with both versions on aarch64 it works for me. > > > > Maybe the filesystem you are using is different? I'm using ext4. Or is the kernel config not set up quite right? Maybe old kernel? > > The root file system is a squashfs, but the data is stored on ext4. > > We are using the latest vanilla Linux kernel 6.1.52. > > It seems to hang here: > WARN[2023-09-08T06:17:38.515218691Z] waiting for response from boltdb > open plugin=bolt > > From what I can tell it's some IPC mechanism or similar which is > hanging. > > I can also reproduce it on x86-64. I guess it must be unique to our > operating system configuration in some way... Best, Christian
Hi Christian, Finally came around looking into this a bit more closely. It seems that containerd just didn't send the sd notification READY=1. It seems that the problem is caused by the CRI plug-in. The plug-in reports the following error: time="2023-09-24T21:43:20.958509027Z" level=warning msg="failed to load plugin io.containerd.grpc.v1.cri" error="failed to create CRI service: failed to create cni conf monitor for default: failed to create the parent of the cni conf dir=/etc/cni: mkdir /etc/cni: read-only We are running containerd on a read-only squashfs. I guess this explains also why you haven't seen this issue. After disabling the plug-in via /etc/containerd/config.toml disabled_plugins = [ "cri" ], the issue disappeared (containerd service starts up correctly and prints the message "containerd successfully booted in ...s" Checking the changelog of v1.6.22 brings up this change: https://github.com/containerd/containerd/pull/8826 The error above was already present before, so I didn't give it much attention. But I guess the change to use atomicfile really changed the behavior and now seems to cause the hang on startup. In any case, from my point of view, nothing which needs to be addressed in Buildroot. Best regards, Stefan On 2023-09-15 23:48, Christian Stewart wrote: > Hi Stefan, > > Did you find a solution to this? Just following up. > > On Thu, Sep 7, 2023 at 11:21 PM Stefan Agner <stefan@agner.ch> wrote: >> > I hate to say "works on my machine," but with both versions on aarch64 it works for me. >> > >> > Maybe the filesystem you are using is different? I'm using ext4. Or is the kernel config not set up quite right? Maybe old kernel? >> >> The root file system is a squashfs, but the data is stored on ext4. >> >> We are using the latest vanilla Linux kernel 6.1.52. >> >> It seems to hang here: >> WARN[2023-09-08T06:17:38.515218691Z] waiting for response from boltdb >> open plugin=bolt >> >> From what I can tell it's some IPC mechanism or similar which is >> hanging. >> >> I can also reproduce it on x86-64. I guess it must be unique to our >> operating system configuration in some way... > > Best, > Christian
>>>>> "Stefan" == Stefan Agner <stefan@agner.ch> writes: > Hi Christian, > Finally came around looking into this a bit more closely. It seems that > containerd just didn't send the sd notification READY=1. > It seems that the problem is caused by the CRI plug-in. The plug-in > reports the following error: > time="2023-09-24T21:43:20.958509027Z" level=warning msg="failed to load > plugin io.containerd.grpc.v1.cri" error="failed to create CRI service: > failed to create cni conf monitor for default: failed to create the > parent of the cni conf dir=/etc/cni: mkdir /etc/cni: read-only > We are running containerd on a read-only squashfs. I guess this explains > also why you haven't seen this issue. > After disabling the plug-in via /etc/containerd/config.toml > disabled_plugins = [ "cri" ], the issue disappeared (containerd service > starts up correctly and prints the message "containerd successfully > booted in ...s" > Checking the changelog of v1.6.22 brings up this change: > https://github.com/containerd/containerd/pull/8826 > The error above was already present before, so I didn't give it much > attention. But I guess the change to use atomicfile really changed the > behavior and now seems to cause the hang on startup. > In any case, from my point of view, nothing which needs to be addressed > in Buildroot. If cri is only needed for when containerd is used with kubernetes, then perhaps we should disable it by default so other people don't need to debug the same issue.
Hi Peter, https://github.com/containerd/containerd/issues/9139#issuecomment-1736477534 On Sat, Sep 30, 2023 at 1:03 AM Peter Korsgaard <peter@korsgaard.com> wrote: > If cri is only needed for when containerd is used with kubernetes, then > perhaps we should disable it by default so other people don't need to > debug the same issue. I think we can safely disable cri on the default config. Best, Christian
On 2023-10-02 20:28, Christian Stewart wrote: > Hi Peter, > > https://github.com/containerd/containerd/issues/9139#issuecomment-1736477534 > > On Sat, Sep 30, 2023 at 1:03 AM Peter Korsgaard <peter@korsgaard.com> wrote: >> If cri is only needed for when containerd is used with kubernetes, then >> perhaps we should disable it by default so other people don't need to >> debug the same issue. > > I think we can safely disable cri on the default config. At that point we can also pass the no_cri build tag. I'll create a PR to do just that. -- Stefan
Stefan, On Wed, Oct 4, 2023, 2:59 AM Stefan Agner <stefan@agner.ch> wrote: > On 2023-10-02 20:28, Christian Stewart wrote: > > Hi Peter, > > > > > https://github.com/containerd/containerd/issues/9139#issuecomment-1736477534 > > > > On Sat, Sep 30, 2023 at 1:03 AM Peter Korsgaard <peter@korsgaard.com> > wrote: > >> If cri is only needed for when containerd is used with kubernetes, then > >> perhaps we should disable it by default so other people don't need to > >> debug the same issue. > > > > I think we can safely disable cri on the default config. > > At that point we can also pass the no_cri build tag. > > I'll create a PR to do just that. > Maybe make it a adjustable config option, since I definitely want to use kubernetes + cri w/ containerd in buildroot. Best, Christian
On 2023-10-04 12:04, Christian Stewart wrote: > Stefan, > > On Wed, Oct 4, 2023, 2:59 AM Stefan Agner <stefan@agner.ch> wrote: > >> On 2023-10-02 20:28, Christian Stewart wrote: >>> Hi Peter, >>> >>> https://github.com/containerd/containerd/issues/9139#issuecomment-1736477534 >>> >>> On Sat, Sep 30, 2023 at 1:03 AM Peter Korsgaard <peter@korsgaard.com> wrote: >>>> If cri is only needed for when containerd is used with kubernetes, then >>>> perhaps we should disable it by default so other people don't need to >>>> debug the same issue. >>> >>> I think we can safely disable cri on the default config. >> >> At that point we can also pass the no_cri build tag. >> >> I'll create a PR to do just that. > > Maybe make it a adjustable config option, since I definitely want to use kubernetes + cri w/ containerd in buildroot. Ok, I'll make a config for that. That said, I would not disable it via config in that case as I would expect someone who opts to build CRI will make sure that the prerequisites (such as writable /etc as it seems) are taken care of. -- Stefan
diff --git a/package/containerd/containerd.hash b/package/containerd/containerd.hash index 345a7b16ff..8404675e30 100644 --- a/package/containerd/containerd.hash +++ b/package/containerd/containerd.hash @@ -1,3 +1,3 @@ # Computed locally -sha256 9452e95455d03a00d78ae0587595d0c18555bae7912068269efa25a724efe713 containerd-1.6.21.tar.gz +sha256 b109aceacc814d7a637ed94ba5ade829cd2642841d03e06971ef124fa3b86899 containerd-1.6.22.tar.gz sha256 4bbe3b885e8cd1907ab4cf9a41e862e74e24b5422297a4f2fe524e6a30ada2b4 LICENSE diff --git a/package/containerd/containerd.mk b/package/containerd/containerd.mk index 83489a3abf..c4a6d854c3 100644 --- a/package/containerd/containerd.mk +++ b/package/containerd/containerd.mk @@ -4,7 +4,7 @@ # ################################################################################ -CONTAINERD_VERSION = 1.6.21 +CONTAINERD_VERSION = 1.6.22 CONTAINERD_SITE = $(call github,containerd,containerd,v$(CONTAINERD_VERSION)) CONTAINERD_LICENSE = Apache-2.0 CONTAINERD_LICENSE_FILES = LICENSE
Bugfixes and updates. https://github.com/containerd/containerd/releases/tag/v1.6.22 Signed-off-by: Christian Stewart <christian@aperture.us> --- package/containerd/containerd.hash | 2 +- package/containerd/containerd.mk | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)