| Message ID | 20191127094837.4045-1-jolsa@kernel.org |
|---|---|
| Headers | show |
| Series | perf/bpftool: Allow to link libbpf dynamically | expand |
On Wed, Nov 27, 2019 at 1:48 AM Jiri Olsa <jolsa@kernel.org> wrote: > > hi, > adding support to link bpftool with libbpf dynamically, > and config change for perf. > > It's now possible to use: > $ make -C tools/bpf/bpftool/ LIBBPF_DYNAMIC=1 > > which will detect libbpf devel package with needed version, > and if found, link it with bpftool. > > It's possible to use arbitrary installed libbpf: > $ make -C tools/bpf/bpftool/ LIBBPF_DYNAMIC=1 LIBBPF_DIR=/tmp/libbpf/ > > I based this change on top of Arnaldo's perf/core, because > it contains libbpf feature detection code as dependency. > It's now also synced with latest bpf-next, so Toke's change > applies correctly. I don't like it. Especially Toke's patch to expose netlink as public and stable libbpf api. bpftools needs to stay tightly coupled with libbpf (and statically linked for that reason). Otherwise libbpf will grow a ton of public api that would have to be stable and will quickly become a burden.
Em Wed, Nov 27, 2019 at 08:37:59AM -0800, Alexei Starovoitov escreveu: > On Wed, Nov 27, 2019 at 1:48 AM Jiri Olsa <jolsa@kernel.org> wrote: > > adding support to link bpftool with libbpf dynamically, > > and config change for perf. > > It's now possible to use: > > $ make -C tools/bpf/bpftool/ LIBBPF_DYNAMIC=1 > > which will detect libbpf devel package with needed version, > > and if found, link it with bpftool. > > It's possible to use arbitrary installed libbpf: > > $ make -C tools/bpf/bpftool/ LIBBPF_DYNAMIC=1 LIBBPF_DIR=/tmp/libbpf/ > > I based this change on top of Arnaldo's perf/core, because > > it contains libbpf feature detection code as dependency. > > It's now also synced with latest bpf-next, so Toke's change > > applies correctly. > I don't like it. > Especially Toke's patch to expose netlink as public and stable libbpf api. > bpftools needs to stay tightly coupled with libbpf (and statically > linked for that reason). > Otherwise libbpf will grow a ton of public api that would have to be stable > and will quickly become a burden. I can relate to that, tools/lib/perf/ is only now getting put in place because of these fears, hopefully the evsel/evlist/cpumap/etc abstractions there have been in use for a long time and will be not be a burden... :-) - Arnaldo
On Wed, Nov 27, 2019 at 8:38 AM Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote: > > On Wed, Nov 27, 2019 at 1:48 AM Jiri Olsa <jolsa@kernel.org> wrote: > > > > hi, > > adding support to link bpftool with libbpf dynamically, > > and config change for perf. > > > > It's now possible to use: > > $ make -C tools/bpf/bpftool/ LIBBPF_DYNAMIC=1 > > > > which will detect libbpf devel package with needed version, > > and if found, link it with bpftool. > > > > It's possible to use arbitrary installed libbpf: > > $ make -C tools/bpf/bpftool/ LIBBPF_DYNAMIC=1 LIBBPF_DIR=/tmp/libbpf/ > > > > I based this change on top of Arnaldo's perf/core, because > > it contains libbpf feature detection code as dependency. > > It's now also synced with latest bpf-next, so Toke's change > > applies correctly. > > I don't like it. > Especially Toke's patch to expose netlink as public and stable libbpf api. > bpftools needs to stay tightly coupled with libbpf (and statically > linked for that reason). > Otherwise libbpf will grow a ton of public api that would have to be stable > and will quickly become a burden. I second that. I'm currently working on adding few more APIs that I'd like to keep unstable for a while, until we have enough real-world usage (and feedback) accumulated, before we stabilize them. With LIBBPF_API and a promise of stable API, we are going to over-stress and over-design APIs, potentially making them either too generic and bloated, or too limited (and thus become deprecated almost at inception time). I'd like to take that pressure off for a super-new and in flux APIs and not hamper the progress. I'm thinking of splitting off those non-stable, sort-of-internal APIs into separate libbpf-experimental.h (or whatever name makes sense), and let those be used only by tools like bpftool, which are only ever statically link against libbpf and are ok with occasional changes to those APIs (which we'll obviously fix in bpftool as well). Pahole seems like another candidate that fits this bill and we might expose some stuff early on to it, if it provides tangible benefits (e.g., BTF dedup speeds ups, etc). Then as APIs mature, we might decide to move them into libbpf.h with LIBBPF_API slapped onto them. Any objections?
On 11/27/19 9:24 PM, Andrii Nakryiko wrote: > On Wed, Nov 27, 2019 at 8:38 AM Alexei Starovoitov > <alexei.starovoitov@gmail.com> wrote: >> On Wed, Nov 27, 2019 at 1:48 AM Jiri Olsa <jolsa@kernel.org> wrote: >>> >>> hi, >>> adding support to link bpftool with libbpf dynamically, >>> and config change for perf. >>> >>> It's now possible to use: >>> $ make -C tools/bpf/bpftool/ LIBBPF_DYNAMIC=1 >>> >>> which will detect libbpf devel package with needed version, >>> and if found, link it with bpftool. >>> >>> It's possible to use arbitrary installed libbpf: >>> $ make -C tools/bpf/bpftool/ LIBBPF_DYNAMIC=1 LIBBPF_DIR=/tmp/libbpf/ >>> >>> I based this change on top of Arnaldo's perf/core, because >>> it contains libbpf feature detection code as dependency. >>> It's now also synced with latest bpf-next, so Toke's change >>> applies correctly. >> >> I don't like it. >> Especially Toke's patch to expose netlink as public and stable libbpf api. >> bpftools needs to stay tightly coupled with libbpf (and statically >> linked for that reason). >> Otherwise libbpf will grow a ton of public api that would have to be stable >> and will quickly become a burden. +1, and would also be out of scope from a BPF library point of view. > I second that. I'm currently working on adding few more APIs that I'd > like to keep unstable for a while, until we have enough real-world > usage (and feedback) accumulated, before we stabilize them. With > LIBBPF_API and a promise of stable API, we are going to over-stress > and over-design APIs, potentially making them either too generic and > bloated, or too limited (and thus become deprecated almost at > inception time). I'd like to take that pressure off for a super-new > and in flux APIs and not hamper the progress. > > I'm thinking of splitting off those non-stable, sort-of-internal APIs > into separate libbpf-experimental.h (or whatever name makes sense), > and let those be used only by tools like bpftool, which are only ever > statically link against libbpf and are ok with occasional changes to > those APIs (which we'll obviously fix in bpftool as well). Pahole > seems like another candidate that fits this bill and we might expose > some stuff early on to it, if it provides tangible benefits (e.g., BTF > dedup speeds ups, etc). > > Then as APIs mature, we might decide to move them into libbpf.h with > LIBBPF_API slapped onto them. Any objections? I don't think adding yet another libbpf_experimental.h makes sense, it feels too much of an invitation to add all sort of random stuff in there. We already do have libbpf.h and libbpf_internal.h, so everything that does not relate to the /stable and public/ API should be moved from libbpf.h into libbpf_internal.h such as the netlink helpers, as one example, and bpftool can use these since in-tree changes also cover the latter just fine. So overall, same page, just reuse/improve libbpf_internal.h instead of a new libbpf_experimental.h. Thanks, Daniel
On Wed, Nov 27, 2019 at 10:22:31PM +0100, Daniel Borkmann wrote: > On 11/27/19 9:24 PM, Andrii Nakryiko wrote: > > On Wed, Nov 27, 2019 at 8:38 AM Alexei Starovoitov > > <alexei.starovoitov@gmail.com> wrote: > > > On Wed, Nov 27, 2019 at 1:48 AM Jiri Olsa <jolsa@kernel.org> wrote: > > > > > > > > hi, > > > > adding support to link bpftool with libbpf dynamically, > > > > and config change for perf. > > > > > > > > It's now possible to use: > > > > $ make -C tools/bpf/bpftool/ LIBBPF_DYNAMIC=1 > > > > > > > > which will detect libbpf devel package with needed version, > > > > and if found, link it with bpftool. > > > > > > > > It's possible to use arbitrary installed libbpf: > > > > $ make -C tools/bpf/bpftool/ LIBBPF_DYNAMIC=1 LIBBPF_DIR=/tmp/libbpf/ > > > > > > > > I based this change on top of Arnaldo's perf/core, because > > > > it contains libbpf feature detection code as dependency. > > > > It's now also synced with latest bpf-next, so Toke's change > > > > applies correctly. > > > > > > I don't like it. > > > Especially Toke's patch to expose netlink as public and stable libbpf api. > > > bpftools needs to stay tightly coupled with libbpf (and statically > > > linked for that reason). > > > Otherwise libbpf will grow a ton of public api that would have to be stable > > > and will quickly become a burden. > > +1, and would also be out of scope from a BPF library point of view. ok, static it is.. ;-) thanks for the feedback, jirka > > > I second that. I'm currently working on adding few more APIs that I'd > > like to keep unstable for a while, until we have enough real-world > > usage (and feedback) accumulated, before we stabilize them. With > > LIBBPF_API and a promise of stable API, we are going to over-stress > > and over-design APIs, potentially making them either too generic and > > bloated, or too limited (and thus become deprecated almost at > > inception time). I'd like to take that pressure off for a super-new > > and in flux APIs and not hamper the progress. > > > > I'm thinking of splitting off those non-stable, sort-of-internal APIs > > into separate libbpf-experimental.h (or whatever name makes sense), > > and let those be used only by tools like bpftool, which are only ever > > statically link against libbpf and are ok with occasional changes to > > those APIs (which we'll obviously fix in bpftool as well). Pahole > > seems like another candidate that fits this bill and we might expose > > some stuff early on to it, if it provides tangible benefits (e.g., BTF > > dedup speeds ups, etc). > > > > Then as APIs mature, we might decide to move them into libbpf.h with > > LIBBPF_API slapped onto them. Any objections? > > I don't think adding yet another libbpf_experimental.h makes sense, it feels > too much of an invitation to add all sort of random stuff in there. We already > do have libbpf.h and libbpf_internal.h, so everything that does not relate to > the /stable and public/ API should be moved from libbpf.h into libbpf_internal.h > such as the netlink helpers, as one example, and bpftool can use these since > in-tree changes also cover the latter just fine. So overall, same page, just > reuse/improve libbpf_internal.h instead of a new libbpf_experimental.h. > > Thanks, > Daniel >
Alexei Starovoitov <alexei.starovoitov@gmail.com> writes: > On Wed, Nov 27, 2019 at 1:48 AM Jiri Olsa <jolsa@kernel.org> wrote: >> >> hi, >> adding support to link bpftool with libbpf dynamically, >> and config change for perf. >> >> It's now possible to use: >> $ make -C tools/bpf/bpftool/ LIBBPF_DYNAMIC=1 >> >> which will detect libbpf devel package with needed version, >> and if found, link it with bpftool. >> >> It's possible to use arbitrary installed libbpf: >> $ make -C tools/bpf/bpftool/ LIBBPF_DYNAMIC=1 LIBBPF_DIR=/tmp/libbpf/ >> >> I based this change on top of Arnaldo's perf/core, because >> it contains libbpf feature detection code as dependency. >> It's now also synced with latest bpf-next, so Toke's change >> applies correctly. > > I don't like it. > Especially Toke's patch to expose netlink as public and stable libbpf > api. Figured you might say that :) > bpftools needs to stay tightly coupled with libbpf (and statically > linked for that reason). > Otherwise libbpf will grow a ton of public api that would have to be stable > and will quickly become a burden. I can see why you don't want to expose the "internal" functions as LIBBPF_API. Doesn't *have* to mean we can't link bpftool dynamically against the .so version of libbpf, though; will see if I can figure out a clean way to do that... -Toke
On Wed, Nov 27, 2019 at 1:49 AM Jiri Olsa <jolsa@kernel.org> wrote: > > hi, > adding support to link bpftool with libbpf dynamically, > and config change for perf. > > It's now possible to use: > $ make -C tools/bpf/bpftool/ LIBBPF_DYNAMIC=1 I wonder what's the motivation behind these changes, though? Why is linking bpftool dynamically with libbpf is necessary and important? They are both developed tightly within kernel repo, so I fail to see what are the huge advantages one can get from linking them dynamically. > > which will detect libbpf devel package with needed version, > and if found, link it with bpftool. > > It's possible to use arbitrary installed libbpf: > $ make -C tools/bpf/bpftool/ LIBBPF_DYNAMIC=1 LIBBPF_DIR=/tmp/libbpf/ > > I based this change on top of Arnaldo's perf/core, because > it contains libbpf feature detection code as dependency. > It's now also synced with latest bpf-next, so Toke's change > applies correctly. > > Also available in: > git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git > libbpf/dyn > > thanks, > jirka > > > --- > Jiri Olsa (2): > perf tools: Allow to specify libbpf install directory > bpftool: Allow to link libbpf dynamically > > Toke Høiland-Jørgensen (1): > libbpf: Export netlink functions used by bpftool > > tools/bpf/bpftool/Makefile | 40 +++++++++++++++++++++++++++++++++++++++- > tools/build/feature/test-libbpf.c | 9 +++++++++ > tools/lib/bpf/libbpf.h | 22 +++++++++++++--------- > tools/lib/bpf/libbpf.map | 7 +++++++ > tools/lib/bpf/nlattr.h | 15 ++++++++++----- > tools/perf/Makefile.config | 27 ++++++++++++++++++++------- > 6 files changed, 98 insertions(+), 22 deletions(-) >
Andrii Nakryiko <andrii.nakryiko@gmail.com> writes: > On Wed, Nov 27, 2019 at 1:49 AM Jiri Olsa <jolsa@kernel.org> wrote: >> >> hi, >> adding support to link bpftool with libbpf dynamically, >> and config change for perf. >> >> It's now possible to use: >> $ make -C tools/bpf/bpftool/ LIBBPF_DYNAMIC=1 > > I wonder what's the motivation behind these changes, though? Why is > linking bpftool dynamically with libbpf is necessary and important? > They are both developed tightly within kernel repo, so I fail to see > what are the huge advantages one can get from linking them > dynamically. Well, all the regular reasons for using dynamic linking (memory usage, binary size, etc). But in particular, the ability to update the libbpf package if there's a serious bug, and have that be picked up by all utilities making use of it. No reason why bpftool should be special in that respect. -Toke
On Mon, Dec 2, 2019 at 10:09 AM Toke Høiland-Jørgensen <toke@redhat.com> wrote: > > Andrii Nakryiko <andrii.nakryiko@gmail.com> writes: > > > On Wed, Nov 27, 2019 at 1:49 AM Jiri Olsa <jolsa@kernel.org> wrote: > >> > >> hi, > >> adding support to link bpftool with libbpf dynamically, > >> and config change for perf. > >> > >> It's now possible to use: > >> $ make -C tools/bpf/bpftool/ LIBBPF_DYNAMIC=1 > > > > I wonder what's the motivation behind these changes, though? Why is > > linking bpftool dynamically with libbpf is necessary and important? > > They are both developed tightly within kernel repo, so I fail to see > > what are the huge advantages one can get from linking them > > dynamically. > > Well, all the regular reasons for using dynamic linking (memory usage, > binary size, etc). bpftool is 327KB with statically linked libbpf. Hardly a huge problem for either binary size or memory usage. CPU instruction cache usage is also hardly a concern for bpftool specifically. > But in particular, the ability to update the libbpf > package if there's a serious bug, and have that be picked up by all > utilities making use of it. I agree, and that works only for utilities linking with libbpf dynamically. For tools that build statically, you'd have to update tools anyways. And if you can update libbpf, you can as well update bpftool at the same time, so I don't think linking bpftool statically with libbpf causes any new problems. > No reason why bpftool should be special in that respect. But I think bpftool is special and we actually want it to be special and tightly coupled to libbpf with sometimes very intimate knowledge of libbpf and access to "hidden" APIs. That allows us to experiment with new stuff that requires use of bpftool (e.g., code generation for BPF programs), without having to expose and seal public APIs. And I don't think it's a problem from the point of code maintenance, because both live in the same repository and are updated "atomically" when new features are added or changed. Beyond superficial binary size worries, I don't see any good reason why we should add more complexity and variables to libbpf and bpftool build processes just to have a "nice to have" option of linking bpftool dynamically with libbpf. > > -Toke >
Em Mon, Dec 02, 2019 at 10:42:53AM -0800, Andrii Nakryiko escreveu: > On Mon, Dec 2, 2019 at 10:09 AM Toke Høiland-Jørgensen <toke@redhat.com> wrote: > > Andrii Nakryiko <andrii.nakryiko@gmail.com> writes: > > > On Wed, Nov 27, 2019 at 1:49 AM Jiri Olsa <jolsa@kernel.org> wrote: > > >> adding support to link bpftool with libbpf dynamically, > > >> and config change for perf. > > >> It's now possible to use: > > >> $ make -C tools/bpf/bpftool/ LIBBPF_DYNAMIC=1 > > > I wonder what's the motivation behind these changes, though? Why is > > > linking bpftool dynamically with libbpf is necessary and important? > > > They are both developed tightly within kernel repo, so I fail to see > > > what are the huge advantages one can get from linking them > > > dynamically. > > Well, all the regular reasons for using dynamic linking (memory usage, > > binary size, etc). > bpftool is 327KB with statically linked libbpf. Hardly a huge problem > for either binary size or memory usage. CPU instruction cache usage is > also hardly a concern for bpftool specifically. > > But in particular, the ability to update the libbpf > > package if there's a serious bug, and have that be picked up by all > > utilities making use of it. > I agree, and that works only for utilities linking with libbpf > dynamically. For tools that build statically, you'd have to update > tools anyways. And if you can update libbpf, you can as well update > bpftool at the same time, so I don't think linking bpftool statically > with libbpf causes any new problems. > > No reason why bpftool should be special in that respect. > But I think bpftool is special and we actually want it to be special > and tightly coupled to libbpf with sometimes very intimate knowledge > of libbpf and access to "hidden" APIs. That allows us to experiment > with new stuff that requires use of bpftool (e.g., code generation for > BPF programs), without having to expose and seal public APIs. And I > don't think it's a problem from the point of code maintenance, because > both live in the same repository and are updated "atomically" when new > features are added or changed. > Beyond superficial binary size worries, I don't see any good reason > why we should add more complexity and variables to libbpf and bpftool > build processes just to have a "nice to have" option of linking > bpftool dynamically with libbpf. s/bpftool/perf/g s/libbpf/libperf/g And I would also agree 8-) - Arnaldo
On Mon, Dec 02, 2019 at 10:42:53AM -0800, Andrii Nakryiko wrote: > On Mon, Dec 2, 2019 at 10:09 AM Toke Høiland-Jørgensen <toke@redhat.com> wrote: > > > > Andrii Nakryiko <andrii.nakryiko@gmail.com> writes: > > > > > On Wed, Nov 27, 2019 at 1:49 AM Jiri Olsa <jolsa@kernel.org> wrote: > > >> > > >> hi, > > >> adding support to link bpftool with libbpf dynamically, > > >> and config change for perf. > > >> > > >> It's now possible to use: > > >> $ make -C tools/bpf/bpftool/ LIBBPF_DYNAMIC=1 > > > > > > I wonder what's the motivation behind these changes, though? Why is > > > linking bpftool dynamically with libbpf is necessary and important? > > > They are both developed tightly within kernel repo, so I fail to see > > > what are the huge advantages one can get from linking them > > > dynamically. > > > > Well, all the regular reasons for using dynamic linking (memory usage, > > binary size, etc). > > bpftool is 327KB with statically linked libbpf. Hardly a huge problem > for either binary size or memory usage. CPU instruction cache usage is > also hardly a concern for bpftool specifically. > > > But in particular, the ability to update the libbpf > > package if there's a serious bug, and have that be picked up by all > > utilities making use of it. > > I agree, and that works only for utilities linking with libbpf > dynamically. For tools that build statically, you'd have to update > tools anyways. And if you can update libbpf, you can as well update > bpftool at the same time, so I don't think linking bpftool statically > with libbpf causes any new problems. it makes difference for us if we need to respin just one library instead of several applications (bpftool and perf at the moment), because of the bug in the library with the Toke's approach we compile some bits of libbpf statically into bpftool, but there's still the official API in the dynamic libbpf that we care about and that could carry on the fix without bpftool respin > > No reason why bpftool should be special in that respect. > > But I think bpftool is special and we actually want it to be special > and tightly coupled to libbpf with sometimes very intimate knowledge > of libbpf and access to "hidden" APIs. That allows us to experiment > with new stuff that requires use of bpftool (e.g., code generation for > BPF programs), without having to expose and seal public APIs. And I > don't think it's a problem from the point of code maintenance, because > both live in the same repository and are updated "atomically" when new > features are added or changed. I thought we solved this by Toke's approach, so there' no need to expose any new/experimental API .. also you guys will probably continue using static linking I guess jirka > > Beyond superficial binary size worries, I don't see any good reason > why we should add more complexity and variables to libbpf and bpftool > build processes just to have a "nice to have" option of linking > bpftool dynamically with libbpf.
On Mon, Dec 2, 2019 at 11:21 AM Jiri Olsa <jolsa@redhat.com> wrote: > > On Mon, Dec 02, 2019 at 10:42:53AM -0800, Andrii Nakryiko wrote: > > On Mon, Dec 2, 2019 at 10:09 AM Toke Høiland-Jørgensen <toke@redhat.com> wrote: > > > > > > Andrii Nakryiko <andrii.nakryiko@gmail.com> writes: > > > > > > > On Wed, Nov 27, 2019 at 1:49 AM Jiri Olsa <jolsa@kernel.org> wrote: > > > >> > > > >> hi, > > > >> adding support to link bpftool with libbpf dynamically, > > > >> and config change for perf. > > > >> > > > >> It's now possible to use: > > > >> $ make -C tools/bpf/bpftool/ LIBBPF_DYNAMIC=1 > > > > > > > > I wonder what's the motivation behind these changes, though? Why is > > > > linking bpftool dynamically with libbpf is necessary and important? > > > > They are both developed tightly within kernel repo, so I fail to see > > > > what are the huge advantages one can get from linking them > > > > dynamically. > > > > > > Well, all the regular reasons for using dynamic linking (memory usage, > > > binary size, etc). > > > > bpftool is 327KB with statically linked libbpf. Hardly a huge problem > > for either binary size or memory usage. CPU instruction cache usage is > > also hardly a concern for bpftool specifically. > > > > > But in particular, the ability to update the libbpf > > > package if there's a serious bug, and have that be picked up by all > > > utilities making use of it. > > > > I agree, and that works only for utilities linking with libbpf > > dynamically. For tools that build statically, you'd have to update > > tools anyways. And if you can update libbpf, you can as well update > > bpftool at the same time, so I don't think linking bpftool statically > > with libbpf causes any new problems. > > it makes difference for us if we need to respin just one library > instead of several applications (bpftool and perf at the moment), > because of the bug in the library > > with the Toke's approach we compile some bits of libbpf statically into > bpftool, but there's still the official API in the dynamic libbpf that > we care about and that could carry on the fix without bpftool respin See my replies on v4 of your patchset. I have doubts this actually works as we hope it works. I also don't see how that is going to work in general. Imagine something like this: static int some_state = 123; LIBBPF_API void set_state(int x) { some_state = x; } int get_state() { return some_state; } If bpftool does: set_state(42); printf("%d\n", get_state()); How is this supposed to work with set_state() coming from libbpf.so, while get_state() being statically linked? Who "owns" memory of `int some_state` -- bpftool or libbpf.so? Can they magically share it? Or rather maybe some_state will be actually two different variables in two different memory regions? And set_state() would be setting one of them, while get_state() would be reading another one? It would be good to test this out. Do you mind checking? > > > > No reason why bpftool should be special in that respect. > > > > But I think bpftool is special and we actually want it to be special > > and tightly coupled to libbpf with sometimes very intimate knowledge > > of libbpf and access to "hidden" APIs. That allows us to experiment > > with new stuff that requires use of bpftool (e.g., code generation for > > BPF programs), without having to expose and seal public APIs. And I > > don't think it's a problem from the point of code maintenance, because > > both live in the same repository and are updated "atomically" when new > > features are added or changed. > > I thought we solved this by Toke's approach, so there' no need > to expose any new/experimental API .. also you guys will probably > continue using static linking I guess > > jirka > > > > > Beyond superficial binary size worries, I don't see any good reason > > why we should add more complexity and variables to libbpf and bpftool > > build processes just to have a "nice to have" option of linking > > bpftool dynamically with libbpf. >
On Mon, Dec 02, 2019 at 11:54:27AM -0800, Andrii Nakryiko wrote: > On Mon, Dec 2, 2019 at 11:21 AM Jiri Olsa <jolsa@redhat.com> wrote: > > > > On Mon, Dec 02, 2019 at 10:42:53AM -0800, Andrii Nakryiko wrote: > > > On Mon, Dec 2, 2019 at 10:09 AM Toke Høiland-Jørgensen <toke@redhat.com> wrote: > > > > > > > > Andrii Nakryiko <andrii.nakryiko@gmail.com> writes: > > > > > > > > > On Wed, Nov 27, 2019 at 1:49 AM Jiri Olsa <jolsa@kernel.org> wrote: > > > > >> > > > > >> hi, > > > > >> adding support to link bpftool with libbpf dynamically, > > > > >> and config change for perf. > > > > >> > > > > >> It's now possible to use: > > > > >> $ make -C tools/bpf/bpftool/ LIBBPF_DYNAMIC=1 > > > > > > > > > > I wonder what's the motivation behind these changes, though? Why is > > > > > linking bpftool dynamically with libbpf is necessary and important? > > > > > They are both developed tightly within kernel repo, so I fail to see > > > > > what are the huge advantages one can get from linking them > > > > > dynamically. > > > > > > > > Well, all the regular reasons for using dynamic linking (memory usage, > > > > binary size, etc). > > > > > > bpftool is 327KB with statically linked libbpf. Hardly a huge problem > > > for either binary size or memory usage. CPU instruction cache usage is > > > also hardly a concern for bpftool specifically. > > > > > > > But in particular, the ability to update the libbpf > > > > package if there's a serious bug, and have that be picked up by all > > > > utilities making use of it. > > > > > > I agree, and that works only for utilities linking with libbpf > > > dynamically. For tools that build statically, you'd have to update > > > tools anyways. And if you can update libbpf, you can as well update > > > bpftool at the same time, so I don't think linking bpftool statically > > > with libbpf causes any new problems. > > > > it makes difference for us if we need to respin just one library > > instead of several applications (bpftool and perf at the moment), > > because of the bug in the library > > > > with the Toke's approach we compile some bits of libbpf statically into > > bpftool, but there's still the official API in the dynamic libbpf that > > we care about and that could carry on the fix without bpftool respin > > See my replies on v4 of your patchset. I have doubts this actually > works as we hope it works. > > I also don't see how that is going to work in general. Imagine > something like this: > > static int some_state = 123; > > LIBBPF_API void set_state(int x) { some_state = x; } > > int get_state() { return some_state; } > > If bpftool does: > > set_state(42); > printf("%d\n", get_state()); > > > How is this supposed to work with set_state() coming from libbpf.so, > while get_state() being statically linked? Who "owns" memory of `int > some_state` -- bpftool or libbpf.so? Can they magically share it? Or > rather maybe some_state will be actually two different variables in > two different memory regions? And set_state() would be setting one of > them, while get_state() would be reading another one? > > It would be good to test this out. Do you mind checking? I think you're right.. sry, I should have checked on this more, there are no relocations for libbpf.so, so it's all statically linked and the libbpf is just in 'needed' libs record.. ugh :-\ jirka
hi, adding support to link bpftool with libbpf dynamically, and config change for perf. It's now possible to use: $ make -C tools/bpf/bpftool/ LIBBPF_DYNAMIC=1 which will detect libbpf devel package with needed version, and if found, link it with bpftool. It's possible to use arbitrary installed libbpf: $ make -C tools/bpf/bpftool/ LIBBPF_DYNAMIC=1 LIBBPF_DIR=/tmp/libbpf/ I based this change on top of Arnaldo's perf/core, because it contains libbpf feature detection code as dependency. It's now also synced with latest bpf-next, so Toke's change applies correctly. Also available in: git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git libbpf/dyn thanks, jirka --- Jiri Olsa (2): perf tools: Allow to specify libbpf install directory bpftool: Allow to link libbpf dynamically Toke Høiland-Jørgensen (1): libbpf: Export netlink functions used by bpftool tools/bpf/bpftool/Makefile | 40 +++++++++++++++++++++++++++++++++++++++- tools/build/feature/test-libbpf.c | 9 +++++++++ tools/lib/bpf/libbpf.h | 22 +++++++++++++--------- tools/lib/bpf/libbpf.map | 7 +++++++ tools/lib/bpf/nlattr.h | 15 ++++++++++----- tools/perf/Makefile.config | 27 ++++++++++++++++++++------- 6 files changed, 98 insertions(+), 22 deletions(-)