Collect the information of per-cpu variables from encoders of worker
threads to the primary encoder.
btf_encoder store per-cpu info separately, not in BTF. Previously, it
merged only BTF types generated by worker threads. So some of the
per-cpu info was missing.
Signed-off-by: Kui-Feng Lee <kuifeng@fb.com>
Tested-by: Andrii Nakryiko <andrii@kernel.org>
Acked-by: Andrii Nakryiko <andrii@kernel.org>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Cc: dwarves@vger.kernel.org
Link: https://lore.kernel.org/r/20220321230837.855572-1-kuifeng@fb.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
This is the same strategy adopted by the bpftool when generating a
vmlinux file using:
$ bpftool btf dump file vmlinux format c > vmlinux.h
Testing it:
$ cat a.c
#include "vmlinux.h"
int main(void)
{
struct saved_context bla;
bla.ds = 1;
return bla.ds + 1;
}
$
$ rm -f a ; make a
cc a.c -o a
$
Now using pahole:
$ pahole --compile > vmlinux.h
$ rm -f a ; make a
$
This still have some issues, like:
$ pahole --compile ../build/allyesconfig/drivers/spi/spi-bitbang.o > spi.h
$ make spi
cc spi.c -o spi
In file included from spi.c:1:
spi.h:7127:38: error: ‘txrx_word’ declared as function returning an array
7127 | u32 (*txrx_word)(struct spi_device *, unsigned int, u32, u8, unsigned int)[4]; /* 200 32 */
| ^~~~~~~~~
make: *** [<builtin>: spi] Error 1
$
The original source code:
/* txrx_word[SPI_MODE_*]() just looks like a shift register */
u32 (*txrx_word[4])(struct spi_device *, unsigned int, u32, u8, unsigned int);
But this misgeneration predates this patch, so shouldn't prevent it from
being merged now, to make progress.
Also there are issues when generating from DWARF, so for now only accept
it if generating from BTF.
Trying to run it with dwarf produces, for now:
$ pahole -F dwarf --compile vmlinux
pahole: --compile currently only works with BTF.
$
Cc: Tony Jones <tonyj@suse.de>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
This reverts commit 005236c3e4.
Dropped since it could not cope with recursive types. A new attempt will
be made on 1.24.
Signed-off-by: Douglas RAILLARD <douglas.raillard@arm.com>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
LLVM patches ([1] for clang, [2] and [3] for BPF backend)
added support for btf_type_tag attributes. The following is
an example:
[$ ~] cat t.c
#define __tag1 __attribute__((btf_type_tag("tag1")))
#define __tag2 __attribute__((btf_type_tag("tag2")))
int __tag1 * __tag1 __tag2 *g __attribute__((section(".data..percpu")));
[$ ~] clang -O2 -g -c t.c
[$ ~] llvm-dwarfdump --debug-info t.o
t.o: file format elf64-x86-64
...
0x0000001e: DW_TAG_variable
DW_AT_name ("g")
DW_AT_type (0x00000033 "int **")
DW_AT_external (true)
DW_AT_decl_file ("/home/yhs/t.c")
DW_AT_decl_line (3)
DW_AT_location (DW_OP_addr 0x0)
0x00000033: DW_TAG_pointer_type
DW_AT_type (0x0000004b "int *")
0x00000038: DW_TAG_LLVM_annotation
DW_AT_name ("btf_type_tag")
DW_AT_const_value ("tag1")
0x00000041: DW_TAG_LLVM_annotation
DW_AT_name ("btf_type_tag")
DW_AT_const_value ("tag2")
0x0000004a: NULL
0x0000004b: DW_TAG_pointer_type
DW_AT_type (0x0000005a "int")
0x00000050: DW_TAG_LLVM_annotation
DW_AT_name ("btf_type_tag")
DW_AT_const_value ("tag1")
0x00000059: NULL
0x0000005a: DW_TAG_base_type
DW_AT_name ("int")
DW_AT_encoding (DW_ATE_signed)
DW_AT_byte_size (0x04)
0x00000061: NULL
From the above example, you can see that DW_TAG_pointer_type may contain
one or more DW_TAG_LLVM_annotation btf_type_tag tags. If
DW_TAG_LLVM_annotation tags are present inside DW_TAG_pointer_type, for
BTF encoding, pahole will need to follow [3] to generate a type chain
like:
var -> ptr -> tag2 -> tag1 -> ptr -> tag1 -> int
This patch implemented dwarf_loader support. If a pointer type contains
DW_TAG_LLVM_annotation tags, a new type btf_type_tag_ptr_type will be
created which will store the pointer tag itself and all
DW_TAG_LLVM_annotation tags. During recoding stage, the type chain will
be formed properly based on the above example.
An option "--skip_encoding_btf_type_tag" is added to disable
this new functionality.
[1] https://reviews.llvm.org/D111199
[2] https://reviews.llvm.org/D113222
[3] https://reviews.llvm.org/D113496
Signed-off-by: Yonghong Song <yhs@fb.com>
Acked-by: Andrii Nakryiko <andrii@kernel.org>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Cc: bpf@vger.kernel.org
Cc: dwarves@vger.kernel.org
Cc: kernel-team@fb.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Remove the global variable and turn it into a member in struct
conf_fprintf, so that it can be used by other parts of the code.
Signed-off-by: Douglas Raillard <douglas.raillard@arm.com>
Cc: dwarves@vger.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Kernel commit ([1]) renamed btf_tag to btf_decl_tag for uapi btf.h and
libbpf api's. The reason is a new clang attribute, btf_type_tag, is
introduced ([2]). Renaming btf_tag to btf_decl_tag makes it easier to
distinghish from btf_type_tag.
I also pulled in latest libbpf repo since it contains renamed libbpf api
function btf__add_decl_tag().
[1] https://lore.kernel.org/bpf/20211012164838.3345699-1-yhs@fb.com/
[2] https://reviews.llvm.org/D111199
Signed-off-by: Yonghong Song <yhs@fb.com>
[ Minor fixups to cope with --skip_missing ]
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Add a --skip_missing option that allows pahole to keep going in case one
of the type passed to -C (e.g. via a file) does not exist.
This is useful for intropsection software such as debugging kernel
modules that can handle various kernel configurations and versions for
which some recently added types are missing. The consumer of the header
becomes responsible of gating the uses of the type with #ifdef
CONFIG_XXX, rather than pahole bailing out on the first unknown type.
Committer testing:
Before:
$ pahole tcp_splice_state,xxfrm_policy_queue,list_head tcp.o
struct tcp_splice_state {
struct pipe_inode_info * pipe; /* 0 8 */
size_t len; /* 8 8 */
unsigned int flags; /* 16 4 */
/* size: 24, cachelines: 1, members: 3 */
/* padding: 4 */
/* last cacheline: 24 bytes */
};
pahole: type 'xxfrm_policy_queue' not found
$
After:
$ pahole --help |& grep skip
--skip=COUNT Skip COUNT input records
--skip_encoding_btf_tag Do not encode TAGs in BTF.
--skip_encoding_btf_vars Do not encode VARs in BTF.
--skip_missing skip missing types passed to -C rather than stop
$ pahole --skip_missing tcp_splice_state,xxfrm_policy_queue,list_head tcp.o
struct tcp_splice_state {
struct pipe_inode_info * pipe; /* 0 8 */
size_t len; /* 8 8 */
unsigned int flags; /* 16 4 */
/* size: 24, cachelines: 1, members: 3 */
/* padding: 4 */
/* last cacheline: 24 bytes */
};
struct list_head {
struct list_head * next; /* 0 8 */
struct list_head * prev; /* 8 8 */
/* size: 16, cachelines: 1, members: 2 */
/* last cacheline: 16 bytes */
};
pahole: type 'xxfrm_policy_queue' not found
$
Signed-off-by: Douglas Raillard <douglas.raillard@arm.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: dwarves@vger.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Parse the DWARF tag DW_TAG_LLVM_annotation. Only record annotations with
btf_tag name which corresponds to btf_tag attributes in C code. Such
information will be used later by the btf_encoder for BTF conversion.
The LLVM implementation only supports btf_tag annotations on
struct/union, func, func parameter and variable ([1]). So we only check
existence of corresponding DW tags in these places.
A flag "--skip_encoding_btf_tag" is introduced if for whatever reason
this feature needs to be disabled.
[1] https://reviews.llvm.org/D106614
Signed-off-by: Yonghong Song <yhs@fb.com>
Acked-by: Andrii Nakryiko <andrii@kernel.org>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Arnaldo Carvalho de Melo <arnaldo.melo@gmail.com>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Cc: bpf@vger.kernel.org
Cc: dwarves@vger.kernel.org
Cc: kernel-team@fb.com
Link: https://lore.kernel.org/r/20210922021326.2287095-1-yhs@fb.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Useful while developing to help in tuning the ptr tables (types, tags,
functions, maybe some more in the future).
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
When using multithreaded DWARF loading we can't really freely access
some tables, as they may grow and lead to stale data accesses generating
segfaults.
So use a type comparision that takes into account just the immutable
information for structs, unions.
This isn't enough to discern if two types with the same name are really
the same, as we need to look at the member types to figure that out.
So if there are types for which member types need to be checked, leave
it for when all CUs were processed and are thus completely immutable to
resort and fully compare such types.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
But this still doesn't cover all types in the kernel, at least not for
btfdiff's needs, which is to have the output of pahole for BTF and DWARF
to be the same, so if we have two types that are ABI equal, it will
still complain if...
$ btfdiff vmlinux
--- /tmp/btfdiff.dwarf.VUXlsB 2021-08-06 18:11:51.371012024 -0300
+++ /tmp/btfdiff.btf.CeZ7hA 2021-08-06 18:11:51.604017029 -0300
@@ -48226,8 +48226,8 @@ struct intel_ir_data {
/* last cacheline: 56 bytes */
};
struct intel_pad_context {
- u32 padctrl0; /* 0 4 */
- u32 padctrl1; /* 4 4 */
+ u32 conf0; /* 0 4 */
+ u32 val; /* 4 4 */
/* size: 8, cachelines: 1, members: 2 */
/* last cacheline: 8 bytes */
$
The name of some members are different :-\ Consider it in the next
patch and possibly add a knob to consider both types equal, i.e. don't
compare member names, just size, number of members and types of pairs of
members (at each offset in both types).
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
In this case we have:
case foo: {
}
case bar:
The fallthru comment has to be _after_ the closing curly brace, fix it
and avoid this warning (from clang, but probably from gcc too):
/var/home/acme/git/pahole/pahole.c:573:40: warning: this statement may fall through [-Wimplicit-fallthrough=]
573 | case DW_TAG_base_type: {
| ^
/var/home/acme/git/pahole/pahole.c:582:17: note: here
582 | case DW_TAG_pointer_type:
| ^~~~
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
To address this clang warning:
/var/home/acme/git/pahole/pahole.c: In function ‘type__instance_read_once’:
/var/home/acme/git/pahole/pahole.c:1933:78: warning: operand of ‘?:’ changes signedness from ‘int’ to ‘uint32_t’ {aka ‘unsigned int’} due to unsignedness of other operand [-Wsign-compare]
1933 | return fread(instance->instance, instance->type->size, 1, fp) != 1 ? -1 : instance->type->size;
Fixes: e3e5a4626c ("pahole: Make sure the header is read only once")
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
It was using &class_names directly while it was also being passed as the
'prototypes' argument, use the argument.
Fixes: 823739b56f ("pahole: Convert class_names into a list of struct prototypes")
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
It should, as its natural to do:
$ pahole --nr_methods -C sock
And have it traverse all functions in all compilation units and show how
many of them have 'struct sock *' as one of its arguments, but more
changes are needed to have this in place and it is easy enough to do:
$ pahole --nr_methods | grep -w sock
$ pahole --nr_methods | grep -w sock
sock 1005
$
And with BTF, its super fast too.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
As we by now already store the 'struct class' it comes from and
class->name is now a string, no point in storing a duplicate name.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
According to Mark Wieelard and as per testing, elfutils' libdw version
must be at least 0.178 for multithreaded DWARF loading.
Check that and emit a warning and then continue using just a single
thread, this allows for asking for multithreading in things like the
Linux Kernel makefiles while still working on older systems, such as
centos:7, where the elfutils version is 0.176.
Mark also provided this info for people using centos:7 (and
equivalents):
''Note that on centos7 if you install centos-release-scl you can get the
various devtoolset packages that do contain newer gcc and elfutils. The
latest are devtoolset-10-gcc (gcc-10.2.1) and devtoolset-10-elfutils-devel
(elfutils-0.182).
After installing you can use them with "scl enable devtoolset-10 bash"
which sets up the environment with the new devtools as default.''
A quick attempt at using a lock around all libdw functions ended up
being a too heavy big hammer, making the multithreaded DWARF loader to
be worse than using just a single thread.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
To ask for sorting output, initially by name.
This is needed in 'btfdiff' to diff the output of 'pahole -F dwarf
--jobs N', where N threads will go on consuming DWARF compile units and
and pretty printing them, producing a non deterministic output.
So we need to sort the output for both BTF and DWARF, and then diff
them.
This is still not enough for some cases where different types have the
same name, things like "usb_priv" that exists in multiple DWARF compile
units, the first processed is "winning", i.e. being the only one
considered.
I have to look how BTF handles this to adopt a similar algorithm and
keep btfdiff usable as a regression test for the BTF and DWARF loader
and the BTF encoder.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Needed to defer calling printing classes to after we have all sorted out
by name with the upcoming 'pahole --sort' option, needed to make it
possible to compare 'pahole -F btf' with 'pahole -F dwarf -j', as the
multithreaded DWARF loader will not have all classes in a deterministic
order. This is needed for 'btfdiff'.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Introduce first parallell DWARF loading, test it, then move on to use it
together with BTF encoding.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Prep work for multithreaded DWARF loading, when there will be concurrent
access to this data structure.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Another simplification made possible by using a plain char string
instead of string_t, that was only needed in the core as prep work
for CTF encoding.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Since we stopped using per-cu obstacks we don't need it. If we ever
want to use it we can do per thread obstacks.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Since we stopped using per-cu obstacks we don't need it. If we ever
want to use it we can do per thread obstacks.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Go on processing CUs till we have everything sorted out, which includes
the --header type.
On a file with DWARF info where the header type was the last to be found
it wasn't being resolved and the tool fails to resolve header variable
references and emits this misleading error message:
⬢[acme@toolbox pahole]$ pahole ~/bin/perf --header=perf_file_header --seek_bytes '$header.data.offset' --size_bytes='$header.data.size' -C 'perf_event_header(sizeof,type,type_enum=perf_event_type)' < perf.data
pahole: --seek_bytes ($header.data.offset) makes reference to --header but it wasn't specified
⬢[acme@toolbox pahole]$
And that 'struct perf_file_header' _is_ in one of the CUs in ~/bin/perf:
⬢[acme@toolbox pahole]$ pahole ~/bin/perf -C perf_file_header
struct perf_file_header {
u64 magic; /* 0 8 */
u64 size; /* 8 8 */
u64 attr_size; /* 16 8 */
struct perf_file_section attrs; /* 24 16 */
struct perf_file_section data; /* 40 16 */
struct perf_file_section event_types; /* 56 16 */
/* --- cacheline 1 boundary (64 bytes) was 8 bytes ago --- */
long unsigned int adds_features[4]; /* 72 32 */
/* size: 104, cachelines: 2, members: 7 */
/* last cacheline: 40 bytes */
};
⬢[acme@toolbox pahole]$
With this fix all the records are printed.
This probably wasn't noticed before because most tests were made with a
~/bin/perf file with BTF information, i.e. just one "CU", so the logic
of deferring the pretty printing till everything gets resolved wasn't
being exercised properly.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
It was unconditionally reading from 'stdin', when a 'fp' is supplied.
Fix this as now we'll stop unconditionally reading from stdin for the
pretty printer.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
We were using this just for the ctf_encoder, that never really got
complete, so ditch it.
For BTF the strings table is done by libbpf, so we don't need it there
either.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Kui-Feng Lee
Arnaldo Carvalho de Melo
Douglas RAILLARD
Yonghong Song