| Message ID | 20240312114750.68295-1-i.maximets@ovn.org |
|---|---|
| State | Accepted |
| Commit | fa0dfa18d5344c82bc8387b4f79f4257a59d6f58 |
| Headers | show |
| Series | [ovs-dev] github: Reduce ASLR entropy to be compatible with asan in llvm 14. | expand |
| Context | Check | Description |
|---|---|---|
| ovsrobot/apply-robot | success | apply and check: success |
| ovsrobot/github-robot-_Build_and_Test | success | github build: passed |
| ovsrobot/intel-ovs-compilation | success | test: success |
On 12 Mar 2024, at 12:47, Ilya Maximets wrote: > Starting with image version 20240310.1.0, GitHub runners are using > 32-bit entropy for ASLR: > > $ sudo sysctl -a | grep vm.mmap.rnd > vm.mmap_rnd_bits = 32 > vm.mmap_rnd_compat_bits = 16 > > This breaks all the asan-enabled builds, because older asan gets > confused by memory mappings and crashes with segmentation fault. > > The issue is fixed in newer releases of llvm: > https://github.com/llvm/llvm-project/commit/fb77ca05ffb4f8e666878f2f6718a9fb4d686839 > https://reviews.llvm.org/D148280 > > But these are not available in Ubuntu 22.04 image. > > This should be fixed by GitHub, but until new images are available > reducing ASLR entropy manually to 28 bits to make builds work. > > Reported-at: https://github.com/actions/runner-images/issues/9491 > Signed-off-by: Ilya Maximets <i.maximets@ovn.org> Thanks for finding the solution to this! Acked-by: Eelco Chaudron <echaudro@redhat.com>
Ilya Maximets <i.maximets@ovn.org> writes: > Starting with image version 20240310.1.0, GitHub runners are using > 32-bit entropy for ASLR: > > $ sudo sysctl -a | grep vm.mmap.rnd > vm.mmap_rnd_bits = 32 > vm.mmap_rnd_compat_bits = 16 > > This breaks all the asan-enabled builds, because older asan gets > confused by memory mappings and crashes with segmentation fault. > > The issue is fixed in newer releases of llvm: > https://github.com/llvm/llvm-project/commit/fb77ca05ffb4f8e666878f2f6718a9fb4d686839 > https://reviews.llvm.org/D148280 > > But these are not available in Ubuntu 22.04 image. > > This should be fixed by GitHub, but until new images are available > reducing ASLR entropy manually to 28 bits to make builds work. > > Reported-at: https://github.com/actions/runner-images/issues/9491 > Signed-off-by: Ilya Maximets <i.maximets@ovn.org> > --- Acked-by: Aaron Conole <aconole@redhat.com> We'll probably need something similar in other projects, too... What a mess.
On 3/12/24 13:16, Eelco Chaudron wrote: > > > On 12 Mar 2024, at 12:47, Ilya Maximets wrote: > >> Starting with image version 20240310.1.0, GitHub runners are using >> 32-bit entropy for ASLR: >> >> $ sudo sysctl -a | grep vm.mmap.rnd >> vm.mmap_rnd_bits = 32 >> vm.mmap_rnd_compat_bits = 16 >> >> This breaks all the asan-enabled builds, because older asan gets >> confused by memory mappings and crashes with segmentation fault. >> >> The issue is fixed in newer releases of llvm: >> https://github.com/llvm/llvm-project/commit/fb77ca05ffb4f8e666878f2f6718a9fb4d686839 >> https://reviews.llvm.org/D148280 >> >> But these are not available in Ubuntu 22.04 image. >> >> This should be fixed by GitHub, but until new images are available >> reducing ASLR entropy manually to 28 bits to make builds work. >> >> Reported-at: https://github.com/actions/runner-images/issues/9491 >> Signed-off-by: Ilya Maximets <i.maximets@ovn.org> > > Thanks for finding the solution to this! > > Acked-by: Eelco Chaudron <echaudro@redhat.com> > Acked-by: Dumitru Ceara <dceara@redhat.com> Ilya, if you're OK with it, I can cherry-pick this into the OVN tree too. It applies there too. Here's the commit in my fork: https://github.com/dceara/ovn/commit/7a9fdc76ce18f4081700ebee4aa88708a711f5f5 Thanks, Dumitru
On 3/12/24 14:28, Dumitru Ceara wrote: > On 3/12/24 13:16, Eelco Chaudron wrote: >> >> >> On 12 Mar 2024, at 12:47, Ilya Maximets wrote: >> >>> Starting with image version 20240310.1.0, GitHub runners are using >>> 32-bit entropy for ASLR: >>> >>> $ sudo sysctl -a | grep vm.mmap.rnd >>> vm.mmap_rnd_bits = 32 >>> vm.mmap_rnd_compat_bits = 16 >>> >>> This breaks all the asan-enabled builds, because older asan gets >>> confused by memory mappings and crashes with segmentation fault. >>> >>> The issue is fixed in newer releases of llvm: >>> https://github.com/llvm/llvm-project/commit/fb77ca05ffb4f8e666878f2f6718a9fb4d686839 >>> https://reviews.llvm.org/D148280 >>> >>> But these are not available in Ubuntu 22.04 image. >>> >>> This should be fixed by GitHub, but until new images are available >>> reducing ASLR entropy manually to 28 bits to make builds work. >>> >>> Reported-at: https://github.com/actions/runner-images/issues/9491 >>> Signed-off-by: Ilya Maximets <i.maximets@ovn.org> >> >> Thanks for finding the solution to this! >> >> Acked-by: Eelco Chaudron <echaudro@redhat.com> >> > > Acked-by: Dumitru Ceara <dceara@redhat.com> > > Ilya, if you're OK with it, I can cherry-pick this into the OVN tree > too. It applies there too. Here's the commit in my fork: > > https://github.com/dceara/ovn/commit/7a9fdc76ce18f4081700ebee4aa88708a711f5f5 Yeah, sure. Feel free to take it. Best regards, Ilya Maximets.
On 3/12/24 13:16, Eelco Chaudron wrote: > > > On 12 Mar 2024, at 12:47, Ilya Maximets wrote: > >> Starting with image version 20240310.1.0, GitHub runners are using >> 32-bit entropy for ASLR: >> >> $ sudo sysctl -a | grep vm.mmap.rnd >> vm.mmap_rnd_bits = 32 >> vm.mmap_rnd_compat_bits = 16 >> >> This breaks all the asan-enabled builds, because older asan gets >> confused by memory mappings and crashes with segmentation fault. >> >> The issue is fixed in newer releases of llvm: >> https://github.com/llvm/llvm-project/commit/fb77ca05ffb4f8e666878f2f6718a9fb4d686839 >> https://reviews.llvm.org/D148280 >> >> But these are not available in Ubuntu 22.04 image. >> >> This should be fixed by GitHub, but until new images are available >> reducing ASLR entropy manually to 28 bits to make builds work. >> >> Reported-at: https://github.com/actions/runner-images/issues/9491 >> Signed-off-by: Ilya Maximets <i.maximets@ovn.org> > > Thanks for finding the solution to this! > > Acked-by: Eelco Chaudron <echaudro@redhat.com> > Thanks, Eelco, Dumitru and Aaron. Applied to all branches. Best regards, Ilya Maximets.
On 3/12/24 15:12, Ilya Maximets wrote: > On 3/12/24 13:16, Eelco Chaudron wrote: >> >> >> On 12 Mar 2024, at 12:47, Ilya Maximets wrote: >> >>> Starting with image version 20240310.1.0, GitHub runners are using >>> 32-bit entropy for ASLR: >>> >>> $ sudo sysctl -a | grep vm.mmap.rnd >>> vm.mmap_rnd_bits = 32 >>> vm.mmap_rnd_compat_bits = 16 >>> >>> This breaks all the asan-enabled builds, because older asan gets >>> confused by memory mappings and crashes with segmentation fault. >>> >>> The issue is fixed in newer releases of llvm: >>> https://github.com/llvm/llvm-project/commit/fb77ca05ffb4f8e666878f2f6718a9fb4d686839 >>> https://reviews.llvm.org/D148280 >>> >>> But these are not available in Ubuntu 22.04 image. >>> >>> This should be fixed by GitHub, but until new images are available >>> reducing ASLR entropy manually to 28 bits to make builds work. >>> >>> Reported-at: https://github.com/actions/runner-images/issues/9491 >>> Signed-off-by: Ilya Maximets <i.maximets@ovn.org> >> >> Thanks for finding the solution to this! >> >> Acked-by: Eelco Chaudron <echaudro@redhat.com> >> > > Thanks, Eelco, Dumitru and Aaron. > > Applied to all branches. > Also applied to the OVN tree to main and all branches down to 22.03. Regards, Dumitru
diff --git a/.github/workflows/build-and-test.yml b/.github/workflows/build-and-test.yml index fc7558148..6f5139304 100644 --- a/.github/workflows/build-and-test.yml +++ b/.github/workflows/build-and-test.yml @@ -238,6 +238,14 @@ jobs: if: matrix.m32 != '' run: sudo apt install -y gcc-multilib + - name: Reduce ASLR entropy + if: matrix.sanitizers != '' + # Asan in llvm 14 provided in ubuntu-22.04 is incompatible with + # high-entropy ASLR configured in much newer kernels that GitHub + # runners are using leading to random crashes: + # https://github.com/actions/runner-images/issues/9491 + run: sudo sysctl -w vm.mmap_rnd_bits=28 + - name: prepare run: ./.ci/linux-prepare.sh
Starting with image version 20240310.1.0, GitHub runners are using 32-bit entropy for ASLR: $ sudo sysctl -a | grep vm.mmap.rnd vm.mmap_rnd_bits = 32 vm.mmap_rnd_compat_bits = 16 This breaks all the asan-enabled builds, because older asan gets confused by memory mappings and crashes with segmentation fault. The issue is fixed in newer releases of llvm: https://github.com/llvm/llvm-project/commit/fb77ca05ffb4f8e666878f2f6718a9fb4d686839 https://reviews.llvm.org/D148280 But these are not available in Ubuntu 22.04 image. This should be fixed by GitHub, but until new images are available reducing ASLR entropy manually to 28 bits to make builds work. Reported-at: https://github.com/actions/runner-images/issues/9491 Signed-off-by: Ilya Maximets <i.maximets@ovn.org> --- .github/workflows/build-and-test.yml | 8 ++++++++ 1 file changed, 8 insertions(+)