| Message ID | 6ab1ebdce8bb87f50f68bf1c57ecf988345c2330.1675873210.git.yann.morin.1998@free.fr |
|---|---|
| State | Changes Requested |
| Headers | show |
| Series | package/dmalloc: cleanups and ssp workaround (branch yem/dmalloc) | expand |
>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes: > dmalloc directly calls into $(LD) to generate a shared library our of > the static one. > To detet what command it should run, ./configure tries various > incantations of ld with various command line options until one does not > fail. One of those is (basically): > $(LD) --whole-archive -o contest.o.t contest.a > This makes ./configure conclude what the command to link a shared > library in the Makefile should be, and thus stores that in a variable: > shlinkargs='$(LD) --whole-archive -o $@' > ... which is then AC_SUBST()ed into Makefile.in with a rule like: > $(SHLIB): $(LIBRARY) > @shlinkargs@ $(LIRARY) > which once substiuted, gives: > $(SHLIB): $(LIBRARY) > $(LD) --whole-archive -o $@ $(LIRARY) > However, when SSP is enabled, the __stack_chk_fail_local and co symbols > are provided by additional libraries or object files, and that is the > responsibility of gcc to pass those when linking. But as dmalloc > directly calls ld, it misses those. > Changing dmalloc to use $(CC) is not trivial, however. > First, we can't pass LD=$(TARGET_CC), otherwise the whole package > explodes [0]: indeed --whole-archive is unknown to gcc, so it must be > passed as -Wl,--whole archive instead. So we'd need to add a new test > that uses $(CC), like so: > $(CC) -Wl,--whole-archive -o contest.o.t contest.a > However, in that case, gcc does pass additional libs/objs (like, for > eample, the SSP ones) to the linker. But then those are also inc;luded > in the whole-archive section. This causes the linker to add all the > symbols form those libs/objs, even those not needed for SSP; on some > archs, like PPC, that may require floating point symbols (__muldiv3 et > al.) which are in another library, and thus the linked can't find them. > The proper solution wouild be to add -Wl,--no-whole-archive, but that > would have to be added _after_ the library we want to link, i.e.e we > should be able to evntually run: > $(CC) -Wl,--whole-archive -o $@ $(LIRARY) -Wl,--no-whole-archive > That would require that we introduce a new variable that is added > _afgter_ the $(LIBRARY), e.g. @shlinkargs_post@ or so... > This is a bigger endeavour than we want to pursue... > Since dmalloc is a debugging utility, it is not supposed to go into > production builds, and during debugging, it would not be surprising that > it needs to poke around arrays to debug them. > So, we go the easier route: disable SSP altogether. > [0] with lots of nce colors, but that's not the point, is it? > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> > Cc: Fabrice Fontaine <fontaine.fabrice@gmail.com> Committed to 2022.11.x and 2022.02.x, thanks.
diff --git a/package/dmalloc/dmalloc.mk b/package/dmalloc/dmalloc.mk index 6b90f810b2..5bd6691725 100644 --- a/package/dmalloc/dmalloc.mk +++ b/package/dmalloc/dmalloc.mk @@ -14,6 +14,13 @@ DMALLOC_LICENSE_FILES = LICENSE.txt DMALLOC_INSTALL_STAGING = YES DMALLOC_CFLAGS = $(TARGET_CFLAGS) +# dmalloc uses $(LD) to link, and thus misses the object files or libs that +# are needed to provide the __stack_chk_fail_local and co. symbols. Changing +# to use $(CC) is really more complex that we'd like. Since dmalloc is +# involved in debugging memory allocation, it is not expected to be a +# production library, so we do not care that much that it has SSP. +DMALLOC_CFLAGS += -fno-stack-protector + ifeq ($(BR2_STATIC_LIBS),y) DMALLOC_CONF_OPTS += --disable-shlib else
dmalloc directly calls into $(LD) to generate a shared library our of the static one. To detet what command it should run, ./configure tries various incantations of ld with various command line options until one does not fail. One of those is (basically): $(LD) --whole-archive -o contest.o.t contest.a This makes ./configure conclude what the command to link a shared library in the Makefile should be, and thus stores that in a variable: shlinkargs='$(LD) --whole-archive -o $@' ... which is then AC_SUBST()ed into Makefile.in with a rule like: $(SHLIB): $(LIBRARY) @shlinkargs@ $(LIRARY) which once substiuted, gives: $(SHLIB): $(LIBRARY) $(LD) --whole-archive -o $@ $(LIRARY) However, when SSP is enabled, the __stack_chk_fail_local and co symbols are provided by additional libraries or object files, and that is the responsibility of gcc to pass those when linking. But as dmalloc directly calls ld, it misses those. Changing dmalloc to use $(CC) is not trivial, however. First, we can't pass LD=$(TARGET_CC), otherwise the whole package explodes [0]: indeed --whole-archive is unknown to gcc, so it must be passed as -Wl,--whole archive instead. So we'd need to add a new test that uses $(CC), like so: $(CC) -Wl,--whole-archive -o contest.o.t contest.a However, in that case, gcc does pass additional libs/objs (like, for eample, the SSP ones) to the linker. But then those are also inc;luded in the whole-archive section. This causes the linker to add all the symbols form those libs/objs, even those not needed for SSP; on some archs, like PPC, that may require floating point symbols (__muldiv3 et al.) which are in another library, and thus the linked can't find them. The proper solution wouild be to add -Wl,--no-whole-archive, but that would have to be added _after_ the library we want to link, i.e.e we should be able to evntually run: $(CC) -Wl,--whole-archive -o $@ $(LIRARY) -Wl,--no-whole-archive That would require that we introduce a new variable that is added _afgter_ the $(LIBRARY), e.g. @shlinkargs_post@ or so... This is a bigger endeavour than we want to pursue... Since dmalloc is a debugging utility, it is not supposed to go into production builds, and during debugging, it would not be surprising that it needs to poke around arrays to debug them. So, we go the easier route: disable SSP altogether. [0] with lots of nce colors, but that's not the point, is it? Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr> Cc: Fabrice Fontaine <fontaine.fabrice@gmail.com> --- package/dmalloc/dmalloc.mk | 7 +++++++ 1 file changed, 7 insertions(+)