diff mbox series

[bpf-next,7/9] libbpf: add BTF writing APIs

Message ID 20200923155436.2117661-8-andriin@fb.com
State Changes Requested
Delegated to: BPF Maintainers
Headers show
Series libbpf: BTF writer APIs | expand

Commit Message

Andrii Nakryiko Sept. 23, 2020, 3:54 p.m. UTC 
Add APIs for appending new BTF types at the end of BTF object.

Each BTF kind has either one API of the form btf__append_<kind>(). For types
that have variable amount of additional items (struct/union, enum, func_proto,
datasec), additional API is provided to emit each such item. E.g., for
emitting a struct, one would use the following sequence of API calls:

btf__append_struct(...);
btf__append_field(...);
...
btf__append_field(...);

Each btf__append_field() will ensure that the last BTF type is of STRUCT or
UNION kind and will automatically increment that type's vlen field.

All the strings are provided as C strings (const char *), not a string offset.
This significantly improves usability of BTF writer APIs. All such strings
will be automatically appended to string section or existing string will be
re-used, if such string was already added previously.

Each API attempts to do all the reasonable validations, like enforcing
non-empty names for entities with required names, proper value bounds, various
bit offset restrictions, etc.

Type ID validation is minimal because it's possible to emit a type that refers
to type that will be emitted later, so libbpf has no way to enforce such
cases. User must be careful to properly emit all the necessary types and
specify type IDs that will be valid in the finally generated BTF.

Each of btf__append_<kind>() APIs return new type ID on success or negative
value on error. APIs like btf__append_field() that emit additional items
return zero on success and negative value on error.

Signed-off-by: Andrii Nakryiko <andriin@fb.com>
---
 tools/lib/bpf/btf.c      | 781 +++++++++++++++++++++++++++++++++++++++
 tools/lib/bpf/btf.h      |  37 ++
 tools/lib/bpf/libbpf.map |  19 +
 3 files changed, 837 insertions(+)

Comments

John Fastabend Sept. 24, 2020, 3:59 p.m. UTC | #1 
Andrii Nakryiko wrote:
> Add APIs for appending new BTF types at the end of BTF object.
> 
> Each BTF kind has either one API of the form btf__append_<kind>(). For types
> that have variable amount of additional items (struct/union, enum, func_proto,
> datasec), additional API is provided to emit each such item. E.g., for
> emitting a struct, one would use the following sequence of API calls:
> 
> btf__append_struct(...);
> btf__append_field(...);
> ...
> btf__append_field(...);
> 
> Each btf__append_field() will ensure that the last BTF type is of STRUCT or
> UNION kind and will automatically increment that type's vlen field.
> 
> All the strings are provided as C strings (const char *), not a string offset.
> This significantly improves usability of BTF writer APIs. All such strings
> will be automatically appended to string section or existing string will be
> re-used, if such string was already added previously.
> 
> Each API attempts to do all the reasonable validations, like enforcing
> non-empty names for entities with required names, proper value bounds, various
> bit offset restrictions, etc.
> 
> Type ID validation is minimal because it's possible to emit a type that refers
> to type that will be emitted later, so libbpf has no way to enforce such
> cases. User must be careful to properly emit all the necessary types and
> specify type IDs that will be valid in the finally generated BTF.
> 
> Each of btf__append_<kind>() APIs return new type ID on success or negative
> value on error. APIs like btf__append_field() that emit additional items
> return zero on success and negative value on error.
> 
> Signed-off-by: Andrii Nakryiko <andriin@fb.com>
> ---

[...]

> +	/* deconstruct BTF, if necessary, and invalidate raw_data */
> +	if (btf_ensure_modifiable(btf))
> +		return -ENOMEM;
> +
> +	sz = sizeof(struct btf_type) + sizeof(int);
> +	t = btf_add_type_mem(btf, sz);
> +	if (!t)
> +		return -ENOMEM;
> +
> +	/* if something goes wrong later, we might end up with extra an string,

nit typo, 'with an extra string'

> +	 * but that shouldn't be a problem, because BTF can't be constructed
> +	 * completely anyway and will most probably be just discarded
> +	 */
> +	name_off = btf__add_str(btf, name);
> +	if (name_off < 0)
> +		return name_off;
> +
> +	t->name_off = name_off;
> +	t->info = btf_type_info(BTF_KIND_INT, 0, 0);
> +	t->size = byte_sz;
> +	/* set INT info, we don't allow setting legacy bit offset/size */
> +	*(__u32 *)(t + 1) = (encoding << 24) | (byte_sz * 8);
> +
> +	err = btf_add_type_idx_entry(btf, btf->hdr->type_len);
> +	if (err)
> +		return err;
> +
> +	btf->hdr->type_len += sz;
> +	btf->hdr->str_off += sz;
> +	btf->nr_types++;
> +	return btf->nr_types;
> +}
> +
Alexei Starovoitov Sept. 25, 2020, 3:55 a.m. UTC | #2 
On Wed, Sep 23, 2020 at 08:54:34AM -0700, Andrii Nakryiko wrote:
> Add APIs for appending new BTF types at the end of BTF object.
> 
> Each BTF kind has either one API of the form btf__append_<kind>(). For types
> that have variable amount of additional items (struct/union, enum, func_proto,
> datasec), additional API is provided to emit each such item. E.g., for
> emitting a struct, one would use the following sequence of API calls:
> 
> btf__append_struct(...);
> btf__append_field(...);
> ...
> btf__append_field(...);

I've just started looking through the diffs. The first thing that struck me
is the name :) Why 'append' instead of 'add' ? The latter is shorter.

Also how would you add anon struct that is within another struct ?
The anon one would have to be added first and then added as a field?
Feels a bit odd that struct/union building doesn't have 'finish' method,
but I guess it can work.
Andrii Nakryiko Sept. 25, 2020, 6:21 a.m. UTC | #3 
On Thu, Sep 24, 2020 at 8:55 PM Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> On Wed, Sep 23, 2020 at 08:54:34AM -0700, Andrii Nakryiko wrote:
> > Add APIs for appending new BTF types at the end of BTF object.
> >
> > Each BTF kind has either one API of the form btf__append_<kind>(). For types
> > that have variable amount of additional items (struct/union, enum, func_proto,
> > datasec), additional API is provided to emit each such item. E.g., for
> > emitting a struct, one would use the following sequence of API calls:
> >
> > btf__append_struct(...);
> > btf__append_field(...);
> > ...
> > btf__append_field(...);
>
> I've just started looking through the diffs. The first thing that struck me
> is the name :) Why 'append' instead of 'add' ? The latter is shorter.

Append is very precise about those types being added at the end. Add
is more ambiguous in that sense and doesn't imply any specific order.
E.g., for btf__add_str() that's suitable, because the order in which
strings are inserted might be different (e.g., if the string is
duplicated). But it's not an "insert" either, so I'm fine with
renaming to "add", if you prefer it.

>
> Also how would you add anon struct that is within another struct ?
> The anon one would have to be added first and then added as a field?
> Feels a bit odd that struct/union building doesn't have 'finish' method,
> but I guess it can work.

That embedded anon struct will be a separate type, then the field
(anonymous or not, that's orthogonal to anonymity of a struct (!))
will refer to that anon struct type by its ID. Anon struct can be
added right before, right after, or in between completely unrelated
types, it doesn't matter to BTF itself as long as all the type IDs
match in the end.

As for the finish method... There wasn't a need so far to have it, as
BTF constantly maintains correct vlen for the "current"
struct/union/func_proto/datasec/enum (anything with extra members).
I've been writing a few more tests than what I posted here (they will
come in the second wave of patches) and converted pahole to this new
API completely. And it does feel pretty nice and natural so far. In
the future we might add something like that, I suppose, that would
allow to do some more validations at the end. But that would be a
non-breaking extension, so I don't want to do it right now.
Alexei Starovoitov Sept. 25, 2020, 3:37 p.m. UTC | #4 
On 9/24/20 11:21 PM, Andrii Nakryiko wrote:
> On Thu, Sep 24, 2020 at 8:55 PM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
>>
>> On Wed, Sep 23, 2020 at 08:54:34AM -0700, Andrii Nakryiko wrote:
>>> Add APIs for appending new BTF types at the end of BTF object.
>>>
>>> Each BTF kind has either one API of the form btf__append_<kind>(). For types
>>> that have variable amount of additional items (struct/union, enum, func_proto,
>>> datasec), additional API is provided to emit each such item. E.g., for
>>> emitting a struct, one would use the following sequence of API calls:
>>>
>>> btf__append_struct(...);
>>> btf__append_field(...);
>>> ...
>>> btf__append_field(...);
>>
>> I've just started looking through the diffs. The first thing that struck me
>> is the name :) Why 'append' instead of 'add' ? The latter is shorter.
> 
> Append is very precise about those types being added at the end. Add
> is more ambiguous in that sense and doesn't imply any specific order.
> E.g., for btf__add_str() that's suitable, because the order in which
> strings are inserted might be different (e.g., if the string is
> duplicated). But it's not an "insert" either, so I'm fine with
> renaming to "add", if you prefer it.

The reason I prefer shorter is to be able to write:
btf__add_var(btf, "my_var", global,
              btf__add_const(btf,
              btf__add_volatile(btf,
              btf__add_ptr(btf,
              btf__add_int(btf, "int", 4, signed))));

In other words the shorter the type the more suitable the api
will be for manual construction of types.
Looks like the api already checks for invalid type_id,
so no need to check validity at every build stage.
Hence it's nice to combine multiple btf__add_*() into single line.

I think C language isn't great for 'constructor' style api.
May be on top of the above api we can add c++-like api?
For example we can define
struct btf_builder {
    struct btf_builder * (*_volatile) (void);
    struct btf_builder * (*_const) (void);
    struct btf_builder * (*_int) (char *name, int sz, int encoding);
    struct btf_builder * (_ptr) (void);
};

and the use it as:
     struct btf_builder *b = btf__create_global_builer(...);

     b->_int("int", 4, singed)
      ->_const()
      ->_volatile()
      ->_ptr()
      ->_var("my_var", global);

Every method will be return its own object (only one such object)
while the actual building will be happening in some 'invisible',
global, and mutex protected place.

>>
>> Also how would you add anon struct that is within another struct ?
>> The anon one would have to be added first and then added as a field?
>> Feels a bit odd that struct/union building doesn't have 'finish' method,
>> but I guess it can work.
> 
> That embedded anon struct will be a separate type, then the field
> (anonymous or not, that's orthogonal to anonymity of a struct (!))
> will refer to that anon struct type by its ID. Anon struct can be
> added right before, right after, or in between completely unrelated
> types, it doesn't matter to BTF itself as long as all the type IDs
> match in the end.
> 
> As for the finish method... There wasn't a need so far to have it, as
> BTF constantly maintains correct vlen for the "current"
> struct/union/func_proto/datasec/enum (anything with extra members).
> I've been writing a few more tests than what I posted here (they will
> come in the second wave of patches) and converted pahole to this new
> API completely. And it does feel pretty nice and natural so far. In
> the future we might add something like that, I suppose, that would
> allow to do some more validations at the end. But that would be a
> non-breaking extension, so I don't want to do it right now.

sure. that's fine.
Also I suspect sooner or later would be good to do at least partial
dedup of types while they're being built.
Instead of passing exact btf_id of 'int' everywhere it would be
human friendlier to say:
   b->_int("int", 4, singed)->_var("my_var")
instead of having extra variable that holds btf_id of 'int' and
use as it:
   u32 btf_id_of_int; /* that is used everywhere where 'int' field of 
var needs to be added */
   b->__var("my_var", btf_id_of_int);

I mean since types are being built the real dedup cannot happen,
but dedup of simple types can happen on the fly.
That will justify 'add' vs 'append' as well.
Just a thought.
Andrii Nakryiko Sept. 25, 2020, 4:53 p.m. UTC | #5 
On Fri, Sep 25, 2020 at 8:37 AM Alexei Starovoitov <ast@fb.com> wrote:
>
> On 9/24/20 11:21 PM, Andrii Nakryiko wrote:
> > On Thu, Sep 24, 2020 at 8:55 PM Alexei Starovoitov
> > <alexei.starovoitov@gmail.com> wrote:
> >>
> >> On Wed, Sep 23, 2020 at 08:54:34AM -0700, Andrii Nakryiko wrote:
> >>> Add APIs for appending new BTF types at the end of BTF object.
> >>>
> >>> Each BTF kind has either one API of the form btf__append_<kind>(). For types
> >>> that have variable amount of additional items (struct/union, enum, func_proto,
> >>> datasec), additional API is provided to emit each such item. E.g., for
> >>> emitting a struct, one would use the following sequence of API calls:
> >>>
> >>> btf__append_struct(...);
> >>> btf__append_field(...);
> >>> ...
> >>> btf__append_field(...);
> >>
> >> I've just started looking through the diffs. The first thing that struck me
> >> is the name :) Why 'append' instead of 'add' ? The latter is shorter.
> >
> > Append is very precise about those types being added at the end. Add
> > is more ambiguous in that sense and doesn't imply any specific order.
> > E.g., for btf__add_str() that's suitable, because the order in which
> > strings are inserted might be different (e.g., if the string is
> > duplicated). But it's not an "insert" either, so I'm fine with
> > renaming to "add", if you prefer it.
>
> The reason I prefer shorter is to be able to write:
> btf__add_var(btf, "my_var", global,
>               btf__add_const(btf,
>               btf__add_volatile(btf,
>               btf__add_ptr(btf,
>               btf__add_int(btf, "int", 4, signed))));

That's an interesting way of using it, I'll give you that :)

Ok, I'll switch to "add" name.

>
> In other words the shorter the type the more suitable the api
> will be for manual construction of types.
> Looks like the api already checks for invalid type_id,
> so no need to check validity at every build stage.
> Hence it's nice to combine multiple btf__add_*() into single line.
>
> I think C language isn't great for 'constructor' style api.
> May be on top of the above api we can add c++-like api?
> For example we can define
> struct btf_builder {
>     struct btf_builder * (*_volatile) (void);
>     struct btf_builder * (*_const) (void);
>     struct btf_builder * (*_int) (char *name, int sz, int encoding);
>     struct btf_builder * (_ptr) (void);
> };
>
> and the use it as:
>      struct btf_builder *b = btf__create_global_builer(...);
>
>      b->_int("int", 4, singed)
>       ->_const()
>       ->_volatile()
>       ->_ptr()
>       ->_var("my_var", global);
>
> Every method will be return its own object (only one such object)
> while the actual building will be happening in some 'invisible',
> global, and mutex protected place.
>
> >>
> >> Also how would you add anon struct that is within another struct ?
> >> The anon one would have to be added first and then added as a field?
> >> Feels a bit odd that struct/union building doesn't have 'finish' method,
> >> but I guess it can work.
> >
> > That embedded anon struct will be a separate type, then the field
> > (anonymous or not, that's orthogonal to anonymity of a struct (!))
> > will refer to that anon struct type by its ID. Anon struct can be
> > added right before, right after, or in between completely unrelated
> > types, it doesn't matter to BTF itself as long as all the type IDs
> > match in the end.
> >
> > As for the finish method... There wasn't a need so far to have it, as
> > BTF constantly maintains correct vlen for the "current"
> > struct/union/func_proto/datasec/enum (anything with extra members).
> > I've been writing a few more tests than what I posted here (they will
> > come in the second wave of patches) and converted pahole to this new
> > API completely. And it does feel pretty nice and natural so far. In
> > the future we might add something like that, I suppose, that would
> > allow to do some more validations at the end. But that would be a
> > non-breaking extension, so I don't want to do it right now.
>
> sure. that's fine.
> Also I suspect sooner or later would be good to do at least partial
> dedup of types while they're being built.
> Instead of passing exact btf_id of 'int' everywhere it would be
> human friendlier to say:
>    b->_int("int", 4, singed)->_var("my_var")
> instead of having extra variable that holds btf_id of 'int' and
> use as it:
>    u32 btf_id_of_int; /* that is used everywhere where 'int' field of
> var needs to be added */
>    b->__var("my_var", btf_id_of_int);
>
> I mean since types are being built the real dedup cannot happen,
> but dedup of simple types can happen on the fly.
> That will justify 'add' vs 'append' as well.
> Just a thought.
>

That's doable, though certainly added complexity. Unless you need to
generate millions of those variables, just appending "int"s many times
and then doing dedup once would be faster and simpler, using just a
touch more memory. But certainly something to keep in mind.
diff mbox series

Patch

diff --git a/tools/lib/bpf/btf.c b/tools/lib/bpf/btf.c
index 4d0e532d7b3d..7d50f626b823 100644
--- a/tools/lib/bpf/btf.c
+++ b/tools/lib/bpf/btf.c
@@ -1311,6 +1311,787 @@  int btf__add_str(struct btf *btf, const char *s)
 	return new_off;
 }
 
+static void *btf_add_type_mem(struct btf *btf, size_t add_sz)
+{
+	return btf_add_mem(&btf->types_data, &btf->types_data_cap, 1,
+			   btf->hdr->type_len, UINT_MAX, add_sz);
+}
+
+static __u32 btf_type_info(int kind, int vlen, int kflag)
+{
+	return (kflag << 31) | (kind << 24) | vlen;
+}
+
+static void btf_type_inc_vlen(struct btf_type *t)
+{
+	t->info = btf_type_info(btf_kind(t), btf_vlen(t) + 1, btf_kflag(t));
+}
+
+/*
+ * Append new BTF_KIND_INT type with:
+ *   - *name* - non-empty, non-NULL type name;
+ *   - *sz* - power-of-2 (1, 2, 4, ..) size of the type, in bytes;
+ *   - encoding is a combination of BTF_INT_SIGNED, BTF_INT_CHAR, BTF_INT_BOOL.
+ * Returns:
+ *   - >0, type ID of newly added BTF type;
+ *   - <0, on error.
+ */
+int btf__append_int(struct btf *btf, const char *name, size_t byte_sz, int encoding)
+{
+	struct btf_type *t;
+	int sz, err, name_off;
+
+	/* non-empty name */
+	if (!name || !name[0])
+		return -EINVAL;
+	/* byte_sz must be power of 2 */
+	if (!byte_sz || (byte_sz & (byte_sz - 1)) || byte_sz > 16)
+		return -EINVAL;
+	if (encoding & ~(BTF_INT_SIGNED | BTF_INT_CHAR | BTF_INT_BOOL))
+		return -EINVAL;
+
+	/* deconstruct BTF, if necessary, and invalidate raw_data */
+	if (btf_ensure_modifiable(btf))
+		return -ENOMEM;
+
+	sz = sizeof(struct btf_type) + sizeof(int);
+	t = btf_add_type_mem(btf, sz);
+	if (!t)
+		return -ENOMEM;
+
+	/* if something goes wrong later, we might end up with extra an string,
+	 * but that shouldn't be a problem, because BTF can't be constructed
+	 * completely anyway and will most probably be just discarded
+	 */
+	name_off = btf__add_str(btf, name);
+	if (name_off < 0)
+		return name_off;
+
+	t->name_off = name_off;
+	t->info = btf_type_info(BTF_KIND_INT, 0, 0);
+	t->size = byte_sz;
+	/* set INT info, we don't allow setting legacy bit offset/size */
+	*(__u32 *)(t + 1) = (encoding << 24) | (byte_sz * 8);
+
+	err = btf_add_type_idx_entry(btf, btf->hdr->type_len);
+	if (err)
+		return err;
+
+	btf->hdr->type_len += sz;
+	btf->hdr->str_off += sz;
+	btf->nr_types++;
+	return btf->nr_types;
+}
+
+/* it's completely legal to append BTF types with type IDs pointing forward to
+ * types that haven't been appended yet, so we only make sure that id looks
+ * sane, we can't guarantee that ID will always be valid
+ */
+static int validate_type_id(int id)
+{
+	if (id < 0 || id > BTF_MAX_NR_TYPES)
+		return -EINVAL;
+	return 0;
+}
+
+/* generic append function for PTR, TYPEDEF, CONST/VOLATILE/RESTRICT */
+static int btf_append_ref_kind(struct btf *btf, int kind, const char *name, int ref_type_id)
+{
+	struct btf_type *t;
+	int sz, name_off = 0, err;
+
+	if (validate_type_id(ref_type_id))
+		return -EINVAL;
+
+	if (btf_ensure_modifiable(btf))
+		return -ENOMEM;
+
+	sz = sizeof(struct btf_type);
+	t = btf_add_type_mem(btf, sz);
+	if (!t)
+		return -ENOMEM;
+
+	if (name && name[0]) {
+		name_off = btf__add_str(btf, name);
+		if (name_off < 0)
+			return name_off;
+	}
+
+	t->name_off = name_off;
+	t->info = btf_type_info(kind, 0, 0);
+	t->type = ref_type_id;
+
+	err = btf_add_type_idx_entry(btf, btf->hdr->type_len);
+	if (err)
+		return err;
+
+	btf->hdr->type_len += sz;
+	btf->hdr->str_off += sz;
+	btf->nr_types++;
+	return btf->nr_types;
+}
+
+/*
+ * Append new BTF_KIND_PTR type with:
+ *   - *ref_type_id* - referenced type ID, it might not exist yet;
+ * Returns:
+ *   - >0, type ID of newly added BTF type;
+ *   - <0, on error.
+ */
+int btf__append_ptr(struct btf *btf, int ref_type_id)
+{
+	return btf_append_ref_kind(btf, BTF_KIND_PTR, NULL, ref_type_id);
+}
+
+/*
+ * Append new BTF_KIND_ARRAY type with:
+ *   - *index_type_id* - type ID of the type describing array index;
+ *   - *elem_type_id* - type ID of the type describing array element;
+ *   - *nr_elems* - the size of the array;
+ * Returns:
+ *   - >0, type ID of newly added BTF type;
+ *   - <0, on error.
+ */
+int btf__append_array(struct btf *btf, int index_type_id, int elem_type_id, __u32 nr_elems)
+{
+	struct btf_type *t;
+	struct btf_array *a;
+	int sz, err;
+
+	if (validate_type_id(index_type_id) || validate_type_id(elem_type_id))
+		return -EINVAL;
+
+	if (btf_ensure_modifiable(btf))
+		return -ENOMEM;
+
+	sz = sizeof(struct btf_type) + sizeof(struct btf_array);
+	t = btf_add_type_mem(btf, sz);
+	if (!t)
+		return -ENOMEM;
+
+	t->name_off = 0;
+	t->info = btf_type_info(BTF_KIND_ARRAY, 0, 0);
+	t->size = 0;
+
+	a = btf_array(t);
+	a->type = elem_type_id;
+	a->index_type = index_type_id;
+	a->nelems = nr_elems;
+
+	err = btf_add_type_idx_entry(btf, btf->hdr->type_len);
+	if (err)
+		return err;
+
+	btf->hdr->type_len += sz;
+	btf->hdr->str_off += sz;
+	btf->nr_types++;
+	return btf->nr_types;
+}
+
+/* generic STRUCT/UNION append function */
+static int btf_append_composite(struct btf *btf, int kind, const char *name, __u32 bytes_sz)
+{
+	struct btf_type *t;
+	int sz, err, name_off = 0;
+
+	if (btf_ensure_modifiable(btf))
+		return -ENOMEM;
+
+	sz = sizeof(struct btf_type);
+	t = btf_add_type_mem(btf, sz);
+	if (!t)
+		return -ENOMEM;
+
+	if (name && name[0]) {
+		name_off = btf__add_str(btf, name);
+		if (name_off < 0)
+			return name_off;
+	}
+
+	/* start out with vlen=0 and no kflag; this will be adjusted when
+	 * adding each member
+	 */
+	t->name_off = name_off;
+	t->info = btf_type_info(kind, 0, 0);
+	t->size = bytes_sz;
+
+	err = btf_add_type_idx_entry(btf, btf->hdr->type_len);
+	if (err)
+		return err;
+
+	btf->hdr->type_len += sz;
+	btf->hdr->str_off += sz;
+	btf->nr_types++;
+	return btf->nr_types;
+}
+
+/*
+ * Append new BTF_KIND_STRUCT type with:
+ *   - *name* - name of the struct, can be NULL or empty for anonymous structs;
+ *   - *byte_sz* - size of the struct, in bytes;
+ *
+ * Struct initially has no fields in it. Fields can be added by
+ * btf__append_field() right after btf__append_struct() succeeds. 
+ *
+ * Returns:
+ *   - >0, type ID of newly added BTF type;
+ *   - <0, on error.
+ */
+int btf__append_struct(struct btf *btf, const char *name, __u32 byte_sz)
+{
+	return btf_append_composite(btf, BTF_KIND_STRUCT, name, byte_sz);
+}
+
+/*
+ * Append new BTF_KIND_UNION type with:
+ *   - *name* - name of the union, can be NULL or empty for anonymous union;
+ *   - *byte_sz* - size of the union, in bytes;
+ *
+ * Union initially has no fields in it. Fields can be added by
+ * btf__append_field() right after btf__append_union() succeeds. All fields
+ * should have *bit_offset* of 0.
+ *
+ * Returns:
+ *   - >0, type ID of newly added BTF type;
+ *   - <0, on error.
+ */
+int btf__append_union(struct btf *btf, const char *name, __u32 byte_sz)
+{
+	return btf_append_composite(btf, BTF_KIND_UNION, name, byte_sz);
+}
+
+/*
+ * Append new field for the current STRUCT/UNION type with:
+ *   - *name* - name of the field, can be NULL or empty for anonymous field;
+ *   - *type_id* - type ID for the type describing field type;
+ *   - *bit_offset* - bit offset of the start of the field within struct/union;
+ *   - *bit_size* - bit size of a bitfield, 0 for non-bitfield fields;
+ * Returns:
+ *   -  0, on success;
+ *   - <0, on error.
+ */
+int btf__append_field(struct btf *btf, const char *name, int type_id,
+		      __u32 bit_offset, __u32 bit_size)
+{
+	struct btf_type *t;
+	struct btf_member *m;
+	bool is_bitfield;
+	int sz, name_off = 0;
+
+	/* last type should be union/struct */
+	if (btf->nr_types == 0)
+		return -EINVAL;
+	t = btf_type_by_id(btf, btf->nr_types);
+	if (!btf_is_composite(t))
+		return -EINVAL;
+
+	if (validate_type_id(type_id))
+		return -EINVAL;
+	/* best-effort bit field offset/size enforcement */
+	is_bitfield = bit_size || (bit_offset % 8 != 0);
+	if (is_bitfield && (bit_size == 0 || bit_size > 255 || bit_offset > 0xffffff))
+		return -EINVAL;
+
+	/* only offset 0 is allowed for unions */
+	if (btf_is_union(t) && bit_offset)
+		return -EINVAL;
+
+	/* decompose and invalidate raw data */
+	if (btf_ensure_modifiable(btf))
+		return -ENOMEM;
+
+	sz = sizeof(struct btf_member);
+	m = btf_add_type_mem(btf, sz);
+	if (!m)
+		return -ENOMEM;
+
+	if (name && name[0]) {
+		name_off = btf__add_str(btf, name);
+		if (name_off < 0)
+			return name_off;
+	}
+
+	m->name_off = name_off;
+	m->type = type_id;
+	m->offset = bit_offset | (bit_size << 24);
+
+	/* btf_add_type_mem can invalidate t pointer */
+	t = btf_type_by_id(btf, btf->nr_types);
+	/* update parent type's vlen and kflag */
+	t->info = btf_type_info(btf_kind(t), btf_vlen(t) + 1, is_bitfield || btf_kflag(t));
+
+	btf->hdr->type_len += sz;
+	btf->hdr->str_off += sz;
+	return 0;
+}
+
+/*
+ * Append new BTF_KIND_ENUM type with:
+ *   - *name* - name of the enum, can be NULL or empty for anonymous enums;
+ *   - *byte_sz* - size of the enum, in bytes.
+ *
+ * Enum initially has no enum values in it (and corresponds to enum forward
+ * declaration). Enumerator values can be added by btf__append_enum_value()
+ * immediately after btf__append_enum() succeeds.
+ *
+ * Returns:
+ *   - >0, type ID of newly added BTF type;
+ *   - <0, on error.
+ */
+int btf__append_enum(struct btf *btf, const char *name, __u32 byte_sz)
+{
+	struct btf_type *t;
+	int sz, err, name_off = 0;
+
+	/* byte_sz must be power of 2 */
+	if (!byte_sz || (byte_sz & (byte_sz - 1)) || byte_sz > 8)
+		return -EINVAL;
+
+	if (btf_ensure_modifiable(btf))
+		return -ENOMEM;
+
+	sz = sizeof(struct btf_type);
+	t = btf_add_type_mem(btf, sz);
+	if (!t)
+		return -ENOMEM;
+
+	if (name && name[0]) {
+		name_off = btf__add_str(btf, name);
+		if (name_off < 0)
+			return name_off;
+	}
+
+	/* start out with vlen=0; it will be adjusted when adding enum values */
+	t->name_off = name_off;
+	t->info = btf_type_info(BTF_KIND_ENUM, 0, 0);
+	t->size = byte_sz;
+
+	err = btf_add_type_idx_entry(btf, btf->hdr->type_len);
+	if (err)
+		return err;
+
+	btf->hdr->type_len += sz;
+	btf->hdr->str_off += sz;
+	btf->nr_types++;
+	return btf->nr_types;
+}
+
+/*
+ * Append new enum value for the current ENUM type with:
+ *   - *name* - name of the enumerator value, can't be NULL or empty;
+ *   - *value* - integer value corresponding to enum value *name*;
+ * Returns:
+ *   -  0, on success;
+ *   - <0, on error.
+ */
+int btf__append_enum_value(struct btf *btf, const char *name, __s64 value)
+{
+	struct btf_type *t;
+	struct btf_enum *v;
+	int sz, name_off;
+
+	/* last type should be BTF_KIND_ENUM */
+	if (btf->nr_types == 0)
+		return -EINVAL;
+	t = btf_type_by_id(btf, btf->nr_types);
+	if (!btf_is_enum(t))
+		return -EINVAL;
+
+	/* non-empty name */
+	if (!name || !name[0])
+		return -EINVAL;
+	if (value < INT_MIN || value > UINT_MAX)
+		return -E2BIG;
+
+	/* decompose and invalidate raw data */
+	if (btf_ensure_modifiable(btf))
+		return -ENOMEM;
+
+	sz = sizeof(struct btf_enum);
+	v = btf_add_type_mem(btf, sz);
+	if (!v)
+		return -ENOMEM;
+
+	name_off = btf__add_str(btf, name);
+	if (name_off < 0)
+		return name_off;
+
+	v->name_off = name_off;
+	v->val = value;
+
+	/* update parent type's vlen */
+	t = btf_type_by_id(btf, btf->nr_types);
+	btf_type_inc_vlen(t);
+
+	btf->hdr->type_len += sz;
+	btf->hdr->str_off += sz;
+	return 0;
+}
+
+/*
+ * Append new BTF_KIND_FWD type with:
+ *   - *name*, non-empty/non-NULL name;
+ *   - *fwd_kind*, kind of forward declaration, one of BTF_FWD_STRUCT,
+ *     BTF_FWD_UNION, or BTF_FWD_ENUM;
+ * Returns:
+ *   - >0, type ID of newly added BTF type;
+ *   - <0, on error.
+ */
+int btf__append_fwd(struct btf *btf, const char *name, enum btf_fwd_kind fwd_kind)
+{
+	if (!name || !name[0])
+		return -EINVAL;
+
+	switch (fwd_kind) {
+	case BTF_FWD_STRUCT:
+	case BTF_FWD_UNION: {
+		struct btf_type *t;
+		int id;
+
+		id = btf_append_ref_kind(btf, BTF_KIND_FWD, name, 0);
+		if (id <= 0)
+			return id;
+		t = btf_type_by_id(btf, id);
+		t->info = btf_type_info(BTF_KIND_FWD, 0, fwd_kind == BTF_FWD_UNION);
+		return id;
+	}
+	case BTF_FWD_ENUM:
+		/* enum forward in BTF currently is just an enum with no enum
+		 * values; we also assume a standard 4-byte size for it
+		 */
+		return btf__append_enum(btf, name, sizeof(int));
+	default:
+		return -EINVAL;
+	}
+}
+
+/*
+ * Append new BTF_KING_TYPEDEF type with:
+ *   - *name*, non-empty/non-NULL name;
+ *   - *ref_type_id* - referenced type ID, it might not exist yet;
+ * Returns:
+ *   - >0, type ID of newly added BTF type;
+ *   - <0, on error.
+ */
+int btf__append_typedef(struct btf *btf, const char *name, int ref_type_id)
+{
+	if (!name || !name[0])
+		return -EINVAL;
+
+	return btf_append_ref_kind(btf, BTF_KIND_TYPEDEF, name, ref_type_id);
+}
+
+/*
+ * Append new BTF_KIND_VOLATILE type with:
+ *   - *ref_type_id* - referenced type ID, it might not exist yet;
+ * Returns:
+ *   - >0, type ID of newly added BTF type;
+ *   - <0, on error.
+ */
+int btf__append_volatile(struct btf *btf, int ref_type_id)
+{
+	return btf_append_ref_kind(btf, BTF_KIND_VOLATILE, NULL, ref_type_id);
+}
+
+/*
+ * Append new BTF_KIND_CONST type with:
+ *   - *ref_type_id* - referenced type ID, it might not exist yet;
+ * Returns:
+ *   - >0, type ID of newly added BTF type;
+ *   - <0, on error.
+ */
+int btf__append_const(struct btf *btf, int ref_type_id)
+{
+	return btf_append_ref_kind(btf, BTF_KIND_CONST, NULL, ref_type_id);
+}
+
+/*
+ * Append new BTF_KIND_RESTRICT type with:
+ *   - *ref_type_id* - referenced type ID, it might not exist yet;
+ * Returns:
+ *   - >0, type ID of newly added BTF type;
+ *   - <0, on error.
+ */
+int btf__append_restrict(struct btf *btf, int ref_type_id)
+{
+	return btf_append_ref_kind(btf, BTF_KIND_RESTRICT, NULL, ref_type_id);
+}
+
+/*
+ * Append new BTF_KIND_FUNC type with:
+ *   - *name*, non-empty/non-NULL name;
+ *   - *proto_type_id* - FUNC_PROTO's type ID, it might not exist yet;
+ * Returns:
+ *   - >0, type ID of newly added BTF type;
+ *   - <0, on error.
+ */
+int btf__append_func(struct btf *btf, const char *name,
+		     enum btf_func_linkage linkage, int proto_type_id)
+{
+	int id;
+
+	if (!name || !name[0])
+		return -EINVAL;
+	if (linkage != BTF_FUNC_STATIC && linkage != BTF_FUNC_GLOBAL &&
+	    linkage != BTF_FUNC_EXTERN)
+		return -EINVAL;
+
+	id = btf_append_ref_kind(btf, BTF_KIND_FUNC, name, proto_type_id);
+	if (id > 0) {
+		struct btf_type *t = btf_type_by_id(btf, id);
+
+		t->info = btf_type_info(BTF_KIND_FUNC, linkage, 0);
+	}
+	return id;
+}
+
+/*
+ * Append new BTF_KIND_FUNC_PROTO with:
+ *   - *ret_type_id* - type ID for return result of a function.
+ *
+ * Function prototype initially has no arguments, but they can be added by
+ * btf__append_func_param() one by one, immediately after
+ * btf__append_func_proto() succeeded.
+ *
+ * Returns:
+ *   - >0, type ID of newly added BTF type;
+ *   - <0, on error.
+ */
+int btf__append_func_proto(struct btf *btf, int ret_type_id)
+{
+	struct btf_type *t;
+	int sz, err;
+
+	if (validate_type_id(ret_type_id))
+		return -EINVAL;
+
+	if (btf_ensure_modifiable(btf))
+		return -ENOMEM;
+
+	sz = sizeof(struct btf_type);
+	t = btf_add_type_mem(btf, sz);
+	if (!t)
+		return -ENOMEM;
+
+	/* start out with vlen=0; this will be adjusted when adding enum
+	 * values, if necessary
+	 */
+	t->name_off = 0;
+	t->info = btf_type_info(BTF_KIND_FUNC_PROTO, 0, 0);
+	t->type = ret_type_id;
+
+	err = btf_add_type_idx_entry(btf, btf->hdr->type_len);
+	if (err)
+		return err;
+
+	btf->hdr->type_len += sz;
+	btf->hdr->str_off += sz;
+	btf->nr_types++;
+	return btf->nr_types;
+}
+
+/*
+ * Append new function parameter for current FUNC_PROTO type with:
+ *   - *name* - parameter name, can be NULL or empty;
+ *   - *type_id* - type ID describing the type of the parameter.
+ * Returns:
+ *   -  0, on success;
+ *   - <0, on error.
+ */
+int btf__append_func_param(struct btf *btf, const char *name, int type_id)
+{
+	struct btf_type *t;
+	struct btf_param *p;
+	int sz, name_off = 0;
+
+	if (validate_type_id(type_id))
+		return -EINVAL;
+
+	/* last type should be BTF_KIND_FUNC_PROTO */
+	if (btf->nr_types == 0)
+		return -EINVAL;
+	t = btf_type_by_id(btf, btf->nr_types);
+	if (!btf_is_func_proto(t))
+		return -EINVAL;
+
+	/* decompose and invalidate raw data */
+	if (btf_ensure_modifiable(btf))
+		return -ENOMEM;
+
+	sz = sizeof(struct btf_param);
+	p = btf_add_type_mem(btf, sz);
+	if (!p)
+		return -ENOMEM;
+
+	if (name && name[0]) {
+		name_off = btf__add_str(btf, name);
+		if (name_off < 0)
+			return name_off;
+	}
+
+	p->name_off = name_off;
+	p->type = type_id;
+
+	/* update parent type's vlen */
+	t = btf_type_by_id(btf, btf->nr_types);
+	btf_type_inc_vlen(t);
+
+	btf->hdr->type_len += sz;
+	btf->hdr->str_off += sz;
+	return 0;
+}
+
+/*
+ * Append new BTF_KIND_VAR type with:
+ *   - *name* - non-empty/non-NULL name;
+ *   - *linkage* - variable linkage, one of BTF_VAR_STATIC,
+ *     BTF_VAR_GLOBAL_ALLOCATED, or BTF_VAR_GLOBAL_EXTERN;
+ *   - *type_id* - type ID of the type describing the type of the variable.
+ * Returns:
+ *   - >0, type ID of newly added BTF type;
+ *   - <0, on error.
+ */
+int btf__append_var(struct btf *btf, const char *name, int linkage, int type_id)
+{
+	struct btf_type *t;
+	struct btf_var *v;
+	int sz, err, name_off;
+
+	/* non-empty name */
+	if (!name || !name[0])
+		return -EINVAL;
+	if (linkage != BTF_VAR_STATIC && linkage != BTF_VAR_GLOBAL_ALLOCATED &&
+	    linkage != BTF_VAR_GLOBAL_EXTERN)
+		return -EINVAL;
+	if (validate_type_id(type_id))
+		return -EINVAL;
+
+	/* deconstruct BTF, if necessary, and invalidate raw_data */
+	if (btf_ensure_modifiable(btf))
+		return -ENOMEM;
+
+	sz = sizeof(struct btf_type) + sizeof(struct btf_var);
+	t = btf_add_type_mem(btf, sz);
+	if (!t)
+		return -ENOMEM;
+
+	name_off = btf__add_str(btf, name);
+	if (name_off < 0)
+		return name_off;
+
+	t->name_off = name_off;
+	t->info = btf_type_info(BTF_KIND_VAR, 0, 0);
+	t->type = type_id;
+
+	v = btf_var(t);
+	v->linkage = linkage;
+
+	err = btf_add_type_idx_entry(btf, btf->hdr->type_len);
+	if (err)
+		return err;
+
+	btf->hdr->type_len += sz;
+	btf->hdr->str_off += sz;
+	btf->nr_types++;
+	return btf->nr_types;
+}
+
+/*
+ * Append new BTF_KIND_DATASEC type with:
+ *   - *name* - non-empty/non-NULL name;
+ *   - *byte_sz* - data section size, in bytes.
+ *
+ * Data section is initially empty. Variables info can be added with
+ * btf__append_datasec_var_info() calls, after btf__append_datasec() succeeds.
+ *
+ * Returns:
+ *   - >0, type ID of newly added BTF type;
+ *   - <0, on error.
+ */
+int btf__append_datasec(struct btf *btf, const char *name, __u32 byte_sz)
+{
+	struct btf_type *t;
+	int sz, err, name_off;
+
+	/* non-empty name */
+	if (!name || !name[0])
+		return -EINVAL;
+
+	if (btf_ensure_modifiable(btf))
+		return -ENOMEM;
+
+	sz = sizeof(struct btf_type);
+	t = btf_add_type_mem(btf, sz);
+	if (!t)
+		return -ENOMEM;
+
+	name_off = btf__add_str(btf, name);
+	if (name_off < 0)
+		return name_off;
+
+	/* start with vlen=0, which will be update as var_secinfos are added */
+	t->name_off = name_off;
+	t->info = btf_type_info(BTF_KIND_DATASEC, 0, 0);
+	t->size = byte_sz;
+
+	err = btf_add_type_idx_entry(btf, btf->hdr->type_len);
+	if (err)
+		return err;
+
+	btf->hdr->type_len += sz;
+	btf->hdr->str_off += sz;
+	btf->nr_types++;
+	return btf->nr_types;
+}
+
+/*
+ * Append new data section variable information entry for current DATASEC type:
+ *   - *var_type_id* - type ID, describing type of the variable;
+ *   - *offset* - variable offset within data section, in bytes;
+ *   - *byte_sz* - variable size, in bytes.
+ *
+ * Returns:
+ *   -  0, on success;
+ *   - <0, on error.
+ */
+int btf__append_datasec_var_info(struct btf *btf, int var_type_id, __u32 offset, __u32 byte_sz)
+{
+	struct btf_type *t;
+	struct btf_var_secinfo *v;
+	int sz;
+
+	/* last type should be BTF_KIND_DATASEC */
+	if (btf->nr_types == 0)
+		return -EINVAL;
+	t = btf_type_by_id(btf, btf->nr_types);
+	if (!btf_is_datasec(t))
+		return -EINVAL;
+
+	if (validate_type_id(var_type_id))
+		return -EINVAL;
+
+	/* decompose and invalidate raw data */
+	if (btf_ensure_modifiable(btf))
+		return -ENOMEM;
+
+	sz = sizeof(struct btf_var_secinfo);
+	v = btf_add_type_mem(btf, sz);
+	if (!v)
+		return -ENOMEM;
+
+	v->type = var_type_id;
+	v->offset = offset;
+	v->size = byte_sz;
+
+	/* update parent type's vlen */
+	t = btf_type_by_id(btf, btf->nr_types);
+	btf_type_inc_vlen(t);
+
+	btf->hdr->type_len += sz;
+	btf->hdr->str_off += sz;
+	return 0;
+}
+
 struct btf_ext_sec_setup_param {
 	__u32 off;
 	__u32 len;
diff --git a/tools/lib/bpf/btf.h b/tools/lib/bpf/btf.h
index 830f798b9a77..32dd71eeed38 100644
--- a/tools/lib/bpf/btf.h
+++ b/tools/lib/bpf/btf.h
@@ -74,6 +74,43 @@  LIBBPF_API __u32 btf_ext__line_info_rec_size(const struct btf_ext *btf_ext);
 LIBBPF_API struct btf *libbpf_find_kernel_btf(void);
 
 LIBBPF_API int btf__add_str(struct btf *btf, const char *s);
+LIBBPF_API int btf__append_int(struct btf *btf, const char *name, size_t byte_sz, int encoding);
+LIBBPF_API int btf__append_ptr(struct btf *btf, int ref_type_id);
+LIBBPF_API int btf__append_array(struct btf *btf,
+				 int index_type_id, int elem_type_id, __u32 nr_elems);
+/* struct/union construction APIs */
+LIBBPF_API int btf__append_struct(struct btf *btf, const char *name, __u32 sz);
+LIBBPF_API int btf__append_union(struct btf *btf, const char *name, __u32 sz);
+LIBBPF_API int btf__append_field(struct btf *btf, const char *name, int field_type_id,
+				 __u32 bit_offset, __u32 bit_size);
+
+/* enum construction APIs */
+LIBBPF_API int btf__append_enum(struct btf *btf, const char *name, __u32 bytes_sz);
+LIBBPF_API int btf__append_enum_value(struct btf *btf, const char *name, __s64 value);
+
+enum btf_fwd_kind {
+	BTF_FWD_STRUCT = 0,
+	BTF_FWD_UNION = 1,
+	BTF_FWD_ENUM = 2,
+};
+
+LIBBPF_API int btf__append_fwd(struct btf *btf, const char *name, enum btf_fwd_kind fwd_kind);
+LIBBPF_API int btf__append_typedef(struct btf *btf, const char *name, int ref_type_id);
+LIBBPF_API int btf__append_volatile(struct btf *btf, int ref_type_id);
+LIBBPF_API int btf__append_const(struct btf *btf, int ref_type_id);
+LIBBPF_API int btf__append_restrict(struct btf *btf, int ref_type_id);
+
+/* func and func_proto construction APIs */
+LIBBPF_API int btf__append_func(struct btf *btf, const char *name,
+				enum btf_func_linkage linkage, int proto_type_id);
+LIBBPF_API int btf__append_func_proto(struct btf *btf, int ret_type_id);
+LIBBPF_API int btf__append_func_param(struct btf *btf, const char *name, int type_id);
+
+/* var & datasec construction APIs */
+LIBBPF_API int btf__append_var(struct btf *btf, const char *name, int linkage, int type_id);
+LIBBPF_API int btf__append_datasec(struct btf *btf, const char *name, __u32 byte_sz);
+LIBBPF_API int btf__append_datasec_var_info(struct btf *btf, int var_type_id,
+					    __u32 offset, __u32 byte_sz);
 
 struct btf_dedup_opts {
 	unsigned int dedup_table_size;
diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map
index 180c871b04b6..6ef598472a50 100644
--- a/tools/lib/bpf/libbpf.map
+++ b/tools/lib/bpf/libbpf.map
@@ -305,6 +305,25 @@  LIBBPF_0.2.0 {
 		bpf_prog_bind_map;
 		bpf_program__section_name;
 		btf__add_str;
+		btf__append_array;
+		btf__append_const;
+		btf__append_enum;
+		btf__append_enum_value;
+		btf__append_datasec;
+		btf__append_datasec_var_info;
+		btf__append_field;
+		btf__append_func;
+		btf__append_func_param;
+		btf__append_func_proto;
+		btf__append_fwd;
+		btf__append_int;
+		btf__append_ptr;
+		btf__append_restrict;
+		btf__append_struct;
+		btf__append_typedef;
+		btf__append_union;
+		btf__append_var;
+		btf__append_volatile;
 		btf__new_empty;
 		perf_buffer__buffer_cnt;
 		perf_buffer__buffer_fd;