| Message ID | 20220629102249.1917848-1-i.maximets@ovn.org |
|---|---|
| State | Accepted |
| Commit | 603bc853fb5637d8db0f748c2a9ded87a4b9c67c |
| Headers | show |
| Series | [ovs-dev] dpif-netdev: Fix leak of AVX512 DPIF scratch pad. | expand |
| Context | Check | Description |
|---|---|---|
| ovsrobot/apply-robot | success | apply and check: success |
| ovsrobot/github-robot-_Build_and_Test | fail | github build: failed |
| ovsrobot/intel-ovs-compilation | success | test: success |
On Wed, Jun 29, 2022 at 12:23 PM Ilya Maximets <i.maximets@ovn.org> wrote: > > dp_netdev_input_outer_avx512 allocates a 16KB scratch pad per PMD > thread, but it's never freed. This may cause significant memory > drain in dynamic environments. > > ==4068109==ERROR: LeakSanitizer: detected memory leaks > > Direct leak of 38656 byte(s) in 2 object(s) allocated from: > 0 0xf069fd in posix_memalign (vswitchd/ovs-vswitchd+0xf069fd) > 1 0x1d7ed14 in xmalloc_size_align lib/util.c:254:13 > 2 0x1d7ed14 in xmalloc_pagealign lib/util.c:352:12 > 3 0x2098254 in dp_netdev_input_outer_avx512 lib/dpif-netdev-avx512.c:69:17 > 4 0x191591a in dp_netdev_process_rxq_port lib/dpif-netdev.c:5332:19 > 5 0x190a961 in pmd_thread_main lib/dpif-netdev.c:6963:17 > 6 0x1c4b69a in ovsthread_wrapper lib/ovs-thread.c:422:12 > 7 0x7fd5ea6f1179 in start_thread pthread_create.c > > SUMMARY: AddressSanitizer: 38656 byte(s) leaked in 2 allocation(s). > > Fixes: 9ac84a1a3698 ("dpif-avx512: Add ISA implementation of dpif.") > Signed-off-by: Ilya Maximets <i.maximets@ovn.org> Reviewed-by: David Marchand <david.marchand@redhat.com>
On 29 Jun 2022, at 12:22, Ilya Maximets wrote: > dp_netdev_input_outer_avx512 allocates a 16KB scratch pad per PMD > thread, but it's never freed. This may cause significant memory > drain in dynamic environments. > > ==4068109==ERROR: LeakSanitizer: detected memory leaks > > Direct leak of 38656 byte(s) in 2 object(s) allocated from: > 0 0xf069fd in posix_memalign (vswitchd/ovs-vswitchd+0xf069fd) > 1 0x1d7ed14 in xmalloc_size_align lib/util.c:254:13 > 2 0x1d7ed14 in xmalloc_pagealign lib/util.c:352:12 > 3 0x2098254 in dp_netdev_input_outer_avx512 lib/dpif-netdev-avx512.c:69:17 > 4 0x191591a in dp_netdev_process_rxq_port lib/dpif-netdev.c:5332:19 > 5 0x190a961 in pmd_thread_main lib/dpif-netdev.c:6963:17 > 6 0x1c4b69a in ovsthread_wrapper lib/ovs-thread.c:422:12 > 7 0x7fd5ea6f1179 in start_thread pthread_create.c > > SUMMARY: AddressSanitizer: 38656 byte(s) leaked in 2 allocation(s). > > Fixes: 9ac84a1a3698 ("dpif-avx512: Add ISA implementation of dpif.") > Signed-off-by: Ilya Maximets <i.maximets@ovn.org> > --- Good catch... Do you have some script to run all kinds of sanitizers? Acked-by: Eelco Chaudron <echaudro@redhat.com>
On 6/29/22 13:45, Eelco Chaudron wrote: > > > On 29 Jun 2022, at 12:22, Ilya Maximets wrote: > >> dp_netdev_input_outer_avx512 allocates a 16KB scratch pad per PMD >> thread, but it's never freed. This may cause significant memory >> drain in dynamic environments. >> >> ==4068109==ERROR: LeakSanitizer: detected memory leaks >> >> Direct leak of 38656 byte(s) in 2 object(s) allocated from: >> 0 0xf069fd in posix_memalign (vswitchd/ovs-vswitchd+0xf069fd) >> 1 0x1d7ed14 in xmalloc_size_align lib/util.c:254:13 >> 2 0x1d7ed14 in xmalloc_pagealign lib/util.c:352:12 >> 3 0x2098254 in dp_netdev_input_outer_avx512 lib/dpif-netdev-avx512.c:69:17 >> 4 0x191591a in dp_netdev_process_rxq_port lib/dpif-netdev.c:5332:19 >> 5 0x190a961 in pmd_thread_main lib/dpif-netdev.c:6963:17 >> 6 0x1c4b69a in ovsthread_wrapper lib/ovs-thread.c:422:12 >> 7 0x7fd5ea6f1179 in start_thread pthread_create.c >> >> SUMMARY: AddressSanitizer: 38656 byte(s) leaked in 2 allocation(s). >> >> Fixes: 9ac84a1a3698 ("dpif-avx512: Add ISA implementation of dpif.") >> Signed-off-by: Ilya Maximets <i.maximets@ovn.org> >> --- > > Good catch... Do you have some script to run all kinds of sanitizers? No. For this one I just built with CFLAGS="-msse4.2 -O1 -fno-omit-frame-pointer -fno-common -g -fsanitize=address -fsanitize=undefined" CC=clang And then tried to run with asan environment variables: ASAN_OPTIONS='detect_leaks=1:abort_on_error=true:log_path=asan' UBSAN_OPTIONS='print_stacktrace=1:halt_on_error=true:log_path=ubsan' LeakSanitizer is part of the AddressSanitizer, you just need to ask to detect leaks (we're disabling that by default in tests/atlocal.in). I didn't check, but I guess, you should be able to reproduce the issue by just building with Asan and running: ASAN_OPTIONS='detect_leaks=1:abort_on_error=true:log_path=asan' make check-dpdk > > > Acked-by: Eelco Chaudron <echaudro@redhat.com> >
Hi Ilya, > -----Original Message----- > From: Ilya Maximets <i.maximets@ovn.org> > Sent: Wednesday, June 29, 2022 5:55 PM > To: Eelco Chaudron <echaudro@redhat.com> > Cc: i.maximets@ovn.org; ovs-dev@openvswitch.org; Flavio Leitner > <fbl@sysclose.org>; Amber, Kumar <kumar.amber@intel.com> > Subject: Re: [ovs-dev] [PATCH] dpif-netdev: Fix leak of AVX512 DPIF scratch > pad. > > On 6/29/22 13:45, Eelco Chaudron wrote: > > > > > > On 29 Jun 2022, at 12:22, Ilya Maximets wrote: > > > >> dp_netdev_input_outer_avx512 allocates a 16KB scratch pad per PMD > >> thread, but it's never freed. This may cause significant memory > >> drain in dynamic environments. > >> > >> ==4068109==ERROR: LeakSanitizer: detected memory leaks > >> > >> Direct leak of 38656 byte(s) in 2 object(s) allocated from: > >> 0 0xf069fd in posix_memalign (vswitchd/ovs-vswitchd+0xf069fd) > >> 1 0x1d7ed14 in xmalloc_size_align lib/util.c:254:13 > >> 2 0x1d7ed14 in xmalloc_pagealign lib/util.c:352:12 > >> 3 0x2098254 in dp_netdev_input_outer_avx512 lib/dpif-netdev- > avx512.c:69:17 > >> 4 0x191591a in dp_netdev_process_rxq_port lib/dpif-netdev.c:5332:19 > >> 5 0x190a961 in pmd_thread_main lib/dpif-netdev.c:6963:17 > >> 6 0x1c4b69a in ovsthread_wrapper lib/ovs-thread.c:422:12 > >> 7 0x7fd5ea6f1179 in start_thread pthread_create.c > >> > >> SUMMARY: AddressSanitizer: 38656 byte(s) leaked in 2 allocation(s). > >> > >> Fixes: 9ac84a1a3698 ("dpif-avx512: Add ISA implementation of dpif.") > >> Signed-off-by: Ilya Maximets <i.maximets@ovn.org> > >> --- > > > > Good catch... Do you have some script to run all kinds of sanitizers? > > No. For this one I just built with > CFLAGS="-msse4.2 -O1 -fno-omit-frame-pointer -fno-common -g - > fsanitize=address -fsanitize=undefined" CC=clang > > And then tried to run with asan environment variables: > > ASAN_OPTIONS='detect_leaks=1:abort_on_error=true:log_path=asan' > UBSAN_OPTIONS='print_stacktrace=1:halt_on_error=true:log_path=ubsan' > > LeakSanitizer is part of the AddressSanitizer, you just need to ask to detect > leaks (we're disabling that by default in tests/atlocal.in). > > I didn't check, but I guess, you should be able to reproduce the issue by just > building with Asan and running: > > ASAN_OPTIONS='detect_leaks=1:abort_on_error=true:log_path=asan' make > check-dpdk > > > > > > > Acked-by: Eelco Chaudron <echaudro@redhat.com> > > Acked-by: Kumar Amber <kumar.amber@intel.com> The changes look good to me, and Thanks again for fixing it. Regards Amber
On 6/29/22 15:27, Amber, Kumar wrote: > Hi Ilya, > > >> -----Original Message----- >> From: Ilya Maximets <i.maximets@ovn.org> >> Sent: Wednesday, June 29, 2022 5:55 PM >> To: Eelco Chaudron <echaudro@redhat.com> >> Cc: i.maximets@ovn.org; ovs-dev@openvswitch.org; Flavio Leitner >> <fbl@sysclose.org>; Amber, Kumar <kumar.amber@intel.com> >> Subject: Re: [ovs-dev] [PATCH] dpif-netdev: Fix leak of AVX512 DPIF scratch >> pad. >> >> On 6/29/22 13:45, Eelco Chaudron wrote: >>> >>> >>> On 29 Jun 2022, at 12:22, Ilya Maximets wrote: >>> >>>> dp_netdev_input_outer_avx512 allocates a 16KB scratch pad per PMD >>>> thread, but it's never freed. This may cause significant memory >>>> drain in dynamic environments. >>>> >>>> ==4068109==ERROR: LeakSanitizer: detected memory leaks >>>> >>>> Direct leak of 38656 byte(s) in 2 object(s) allocated from: >>>> 0 0xf069fd in posix_memalign (vswitchd/ovs-vswitchd+0xf069fd) >>>> 1 0x1d7ed14 in xmalloc_size_align lib/util.c:254:13 >>>> 2 0x1d7ed14 in xmalloc_pagealign lib/util.c:352:12 >>>> 3 0x2098254 in dp_netdev_input_outer_avx512 lib/dpif-netdev- >> avx512.c:69:17 >>>> 4 0x191591a in dp_netdev_process_rxq_port lib/dpif-netdev.c:5332:19 >>>> 5 0x190a961 in pmd_thread_main lib/dpif-netdev.c:6963:17 >>>> 6 0x1c4b69a in ovsthread_wrapper lib/ovs-thread.c:422:12 >>>> 7 0x7fd5ea6f1179 in start_thread pthread_create.c >>>> >>>> SUMMARY: AddressSanitizer: 38656 byte(s) leaked in 2 allocation(s). >>>> >>>> Fixes: 9ac84a1a3698 ("dpif-avx512: Add ISA implementation of dpif.") >>>> Signed-off-by: Ilya Maximets <i.maximets@ovn.org> >>>> --- >>> >>> Good catch... Do you have some script to run all kinds of sanitizers? >> >> No. For this one I just built with >> CFLAGS="-msse4.2 -O1 -fno-omit-frame-pointer -fno-common -g - >> fsanitize=address -fsanitize=undefined" CC=clang >> >> And then tried to run with asan environment variables: >> >> ASAN_OPTIONS='detect_leaks=1:abort_on_error=true:log_path=asan' >> UBSAN_OPTIONS='print_stacktrace=1:halt_on_error=true:log_path=ubsan' >> >> LeakSanitizer is part of the AddressSanitizer, you just need to ask to detect >> leaks (we're disabling that by default in tests/atlocal.in). >> >> I didn't check, but I guess, you should be able to reproduce the issue by just >> building with Asan and running: >> >> ASAN_OPTIONS='detect_leaks=1:abort_on_error=true:log_path=asan' make >> check-dpdk >> >>> >>> >>> Acked-by: Eelco Chaudron <echaudro@redhat.com> >>> > Acked-by: Kumar Amber <kumar.amber@intel.com> > > The changes look good to me, and Thanks again for fixing it. Thanks, everyone! Applied and backported down to 2.16. Best regards, Ilya Maximets.
diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index f46b9fe18..d09138f2c 100644 --- a/lib/dpif-netdev.c +++ b/lib/dpif-netdev.c @@ -7523,6 +7523,7 @@ dp_netdev_destroy_pmd(struct dp_netdev_pmd_thread *pmd) seq_destroy(pmd->reload_seq); ovs_mutex_destroy(&pmd->port_mutex); ovs_mutex_destroy(&pmd->bond_mutex); + free(pmd->netdev_input_func_userdata); free(pmd); }
dp_netdev_input_outer_avx512 allocates a 16KB scratch pad per PMD thread, but it's never freed. This may cause significant memory drain in dynamic environments. ==4068109==ERROR: LeakSanitizer: detected memory leaks Direct leak of 38656 byte(s) in 2 object(s) allocated from: 0 0xf069fd in posix_memalign (vswitchd/ovs-vswitchd+0xf069fd) 1 0x1d7ed14 in xmalloc_size_align lib/util.c:254:13 2 0x1d7ed14 in xmalloc_pagealign lib/util.c:352:12 3 0x2098254 in dp_netdev_input_outer_avx512 lib/dpif-netdev-avx512.c:69:17 4 0x191591a in dp_netdev_process_rxq_port lib/dpif-netdev.c:5332:19 5 0x190a961 in pmd_thread_main lib/dpif-netdev.c:6963:17 6 0x1c4b69a in ovsthread_wrapper lib/ovs-thread.c:422:12 7 0x7fd5ea6f1179 in start_thread pthread_create.c SUMMARY: AddressSanitizer: 38656 byte(s) leaked in 2 allocation(s). Fixes: 9ac84a1a3698 ("dpif-avx512: Add ISA implementation of dpif.") Signed-off-by: Ilya Maximets <i.maximets@ovn.org> --- lib/dpif-netdev.c | 1 + 1 file changed, 1 insertion(+)