Instead of relying on error-prone adjustment of bit/byte holes, use
class__find_holes() to re-calculate them after members are moved around.
As part of that change, fix bug with not adjusting bit_offset, when
changing byte_offset.
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Cc: Alexei Starovoitov <ast@fb.com>
Cc: Yonghong Song <yhs@fb.com>
Cc: dwarves@vger.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
pahole --reorganize heavily depends on member's bit_hole and hole fields
to denote bit/byte holes *after* member. Previous commit "dwarves: use
bit sizes and bit/byte hole info in __class__fprintf" changed its
meaning to bit/byte hole *before* member to accomodate possible bit/byte
holes at the beginning of a struct. This change broke reorganization
algorithm, though, which is quite involved and isn't trivially
modifiable to accomodate new semantics.
This patch reverts the meaning of bit_hole and hole, but also introduces
per class pre_bit_hole/pre_hole to record initial bit/byte hole of a
struct. This allows to fix reorg code more easily and still handle
initial holes cases, if at the expense of being not as elegant.
Committer testing:
$ time pahole -F btf --packable vmlinux | sort -nr -k4 | head
bts_ctx 12288 8192 4096
swsusp_info 4096 432 3664
vc_data 792 496 296
pci_dev 2488 2320 168
rcu_state 3392 3240 152
ptr_ring 192 40 152
xdp_sock 960 840 120
zone 1664 1552 112
rcu_data 576 472 104
rcu_node 576 480 96
real 0m0.038s
user 0m0.029s
sys 0m0.017s
$
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Reported-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Alexei Starovoitov <ast@fb.com>
Cc: Yonghong Song <yhs@fb.com>
Cc: dwarves@vger.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
This patch changes __class__fprintf to rely on bit_size/bitfield_size,
calculated in corresponding loaders, instead of trying to guess correct
sizes on its own.
Bit and byte hole information is now stored in current field member and
stores information about holes *before* field. Previously hole
information was stored in previous field and was meant to represent
"holes after a field". Such approach makes it hard to report bit holes
at the very beginning of the struct:
struct s {
int:4; /* this is bit hole, not a field in DWARF/BTF */
int x:8;
};
With this change and previous bitfield calculation/fixup logic fixes,
there are no more discrepancies between DWARF/BTF for allyesconfig
kernel. There are also no more BRAIN FART ALERTS!
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Cc: Alexei Starovoitov <ast@fb.com>
Cc: Mark Wielaard <mark@klomp.org>
Cc: Martin KaFai Lau <kafai@fb.com>
Cc: Yonghong Song <yhs@fb.com>
Cc: dwarves@vger.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
This patch rewrites hole detection logic. Now many crazy combinations of
bitfields and normal fields are handled correctly.
This was tested on allyesconfig kernel and differences before/after were
always in favor of new algorithm.
With subsequent change in next patch, there are no more BRAIN FART
ALERTs for allyesconfig and DWARF and BTF outputs have no discrepanies.
Example:
$ cat test.c
struct s {
short : 4; /* this one is not emitted in DWARF/BTF */
short a : 4;
int x;
int : 10;
int y : 4;
short zz;
short zzz : 4;
long z;
short : 4;
};
int main() {
struct s s;
return 0;
}
$ gcc -g test.c -o test
$ ~/pahole/build/pahole -J test
$ ~/pahole/build/pahole -F dwarf test
struct s {
/* XXX 4 bits hole, try to pack */
short int a:4; /* 0: 8 2 */
/* XXX 8 bits hole, try to pack */
/* XXX 2 bytes hole, try to pack */
int x; /* 4 4 */
/* XXX 10 bits hole, try to pack */
int y:4; /* 8:18 4 */
/* Bitfield combined with next fields */
/* XXX 2 bits hole, try to pack */
short int zz; /* 10 2 */
short int zzz:4; /* 12:12 2 */
/* XXX 12 bits hole, try to pack */
/* XXX 2 bytes hole, try to pack */
long int z; /* 16 8 */
/* size: 32, cachelines: 1, members: 6 */
/* sum members (bits): 124, holes: 2, sum holes: 4 */
/* bit holes: 5, sum bit holes: 36 bits */
/* padding: 8 */
/* last cacheline: 32 bytes */
};
No discrepanies between BTF/DWARF:
$ ../btfdiff test
$
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Alexei Starovoitov <ast@fb.com>
Cc: Mark Wielaard <mark@klomp.org>
Cc: Martin KaFai Lau <kafai@fb.com>
Cc: Yonghong Song <yhs@fb.com>
Cc: dwarves@vger.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
the CTF and BTF loaders come already with the id to use, while the DWARF
loader comes with a Dwarf_Off that needs to be converted into the
ptr_table index.
So keep the cu__add_tag(cu, tag, &id) method to ask ask for the index to
be allocated in the ptr_table and the result to come back via the 'id'
parameter, now a uint32_t and introduce a cu__add_tag_with_id(cu, tag, id)
method to indicate that the 'uint32_t id' is the one to use.
With this we can use a uint32_t for the id both on 32-bit and 64-bit
arches.
Reported-by: Andrii Nakryiko <andriin@fb.com>
Tested-by:Andrii Nakryiko <andrii.nakryiko@gmail.com>
Link: https://lore.kernel.org/bpf/CAEf4Bzb0SpvXdDKMMnUof==kp4Y0AP54bKFjeCzX_AsmDm7k7g@mail.gmail.com/
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
We were using 'long', so that we could return -ENOMEM, but since we need
struct ptr_table members are already uint32_t, meaning we can use the
entire range, make the return be just an int and be just for error
reporting and pass a uint32_t pointer to return the index used for the
new entry.
Reported-by: Andrii Nakryiko <andriin@fb.com>
Tested-by:Andrii Nakryiko <andrii.nakryiko@gmail.com>
Link: https://lore.kernel.org/bpf/CAEf4Bzb0SpvXdDKMMnUof==kp4Y0AP54bKFjeCzX_AsmDm7k7g@mail.gmail.com/
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Existing code base assumes that single CU doesn't have more than 65535
types per each CU, which might be a reasonable assumption for DWARF
data. With BTF, though, all we get is single, potentially huge, CU which
can easily have more than 65k types. For example, this is the case for
allyesconfig version of Linux kernel, which has >200k types.
Due to this assumption, libdwarves and other parts of pahole are using
16-bit counters to iterate over entities within CU. This can cause
infinite loops when iterating BTF data, if there are more than 65535
types. This patch changes non-public variables to use 32-bit integers,
where appropriate.
This still leads to invalid reported data when using BTF loader (due to using
(X & 0xFFFF) type ID, instead of X, when X > 65535) and loading huge files,
but at least it's not stuck in an infinite loop anymore.
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Reported-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Alexei Starovoitov <ast@fb.com>
Cc: Yonghong Song <yhs@fb.com>
Cc: bpf@vger.kernel.org
Cc: dwarves@vger.kernel.org
[ Removed non type ID conversions, for instance for the kind of tag, like in type->namespace.tag.tag, that can remain a uint16_t ]
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Without it we end up with these messages and a zero sized _Float128
type.
$ btfdiff libc-2.28.so.debug
base_type__name_to_size: base_type _Float128
class__fixup_btf_bitfields: unknown base type name "_Float128"!
base_type__name_to_size: base_type _Float128
class__fixup_btf_bitfields: unknown base type name "_Float128"!
base_type__name_to_size: base_type _Float128
class__fixup_btf_bitfields: unknown base type name "_Float128"!
base_type__name_to_size: base_type _Float128
class__fixup_btf_bitfields: unknown base type name "_Float128"!
base_type__name_to_size: base_type _Float128
class__fixup_btf_bitfields: unknown base type name "_Float128"!
base_type__name_to_size: base_type _Float128
class__fixup_btf_bitfields: unknown base type name "_Float128"!
--- /tmp/btfdiff.dwarf.M7kavg 2019-02-26 10:13:06.633184595 -0300
+++ /tmp/btfdiff.btf.vlWt5u 2019-02-26 10:13:06.640184669 -0300
@@ -2142,7 +2149,7 @@ struct ucontext_t {
/* last cacheline: 8 bytes */
};
union ieee854_float128 {
- _Float128 d; /* 0 16 */
+ _Float128 d; /* 0 0 */
struct {
unsigned int mantissa3:32; /* 0: 0 4 */
unsigned int mantissa2:32; /* 4: 0 4 */
After this patch these messages are gone and for pahole's needs we have
enough information, no need to wait for BTF to have explicit support for
floating point base types.
Reported-by: Andrii Nakryiko <andriin@fb.com>
Cc: Alexei Starovoitov <ast@fb.com>
Cc: Yonghong Song <yhs@fb.com>
Link: https://lore.kernel.org/bpf/20190226131156.GA26786@kernel.org/
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
clang seems to generate base type with name "short", instead of "short
in", but it also isn't inconceivable to imagine other compilers
generating just "long" and/or "long long". This patch adds all those
short forms to a list of base type names.
$ cat type_test.c
struct s {
short x1;
long x2;
long long x3;
};
int main() {
struct s s;
return 0;
}
$ clang -g type_test.c -o type_test && ~/local/pahole/build/pahole -JV type_test
File type_test:
[1] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
[2] STRUCT s kind_flag=0 size=24 vlen=3
x1 type_id=3 bits_offset=0
x2 type_id=4 bits_offset=64
x3 type_id=5 bits_offset=128
[3] INT short size=2 bit_offset=0 nr_bits=16 encoding=SIGNED
[4] INT long int size=8 bit_offset=0 nr_bits=64 encoding=SIGNED
[5] INT long long int size=8 bit_offset=0 nr_bits=64 encoding=SIGNED
Before:
$ ~/local/pahole/build/pahole -F btf type_test
base_type__name_to_size: base_type short
class__fixup_btf_bitfields: unknown base type name "short"!
struct s {
short x1; /* 0 0 */
/* XXX 8 bytes hole, try to pack */
long int x2; /* 8 8 */
long long int x3; /* 16 8 */
/* size: 24, cachelines: 1, members: 3 */
/* sum members: 16, holes: 1, sum holes: 8 */
/* last cacheline: 24 bytes */
};
After:
$ ~/local/pahole/build/pahole -F btf type_test
struct s {
short x1; /* 0 2 */
/* XXX 6 bytes hole, try to pack */
long int x2; /* 8 8 */
long long int x3; /* 16 8 */
/* size: 24, cachelines: 1, members: 3 */
/* sum members: 18, holes: 1, sum holes: 6 */
/* last cacheline: 24 bytes */
};
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Alexei Starovoitov <ast@fb.com>
Cc: Yonghong Song <yhs@fb.com>
Cc: bpf@vger.kernel.org
Cc: dwarves@vger.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
So that we don't clash with libbpf's 'struct btf', in time more internal
state now in 'struct btf_elf' will refer to the equivalent internal
state in libbpf's 'struct btf', as they have lots in common.
Requested-by: Andrii Nakryiko <andrii.nakryiko@gmail.com>
Acked-by: Andrii Nakryiko <andrii.nakryiko@gmail.com>
Cc: Alexei Starovoitov <ast@fb.com>
Cc: Martin Lau <kafai@fb.com>
Cc: Yonghong Song <yhs@fb.com>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
In pahole we started showing 'union' tags as well, and those are
represented by 'struct type', so be defensive in class__find_holes(),
checking if the tag is represented with a 'struct class'.
Tested-by: Andrii Nakryiko <andriin@fb.com>
Fixes: 31664d60ad ("pahole: Show tagged enums as well when no class is specified")
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Since tools like 'pahole' now shows unions in addition to structs.
Cc: Matthew Wilcox <willy@infradead.org>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
If we processed at least one file we should return 0 to mean success.
Fixes: 02a456f5f5 ("pahole: Search and use running kernel vmlinux when no file is passed")
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Seems like a nice shortcut for kernel hackers, and one that makes sure
the right vmlinux file is used, as it reads the running kernel build id
from /sys/kernel/notes and then searches the well known path for vmlinux
files:
vmlinux
/boot/vmlinux
/boot/vmlinux-4.14.0+
/usr/lib/debug/boot/vmlinux-`uname -r`
/lib/modules/`uname -r`/build/vmlinux
/usr/lib/debug/lib/modules/`uname -r`/vmlinux
/usr/lib/debug/boot/vmlinux-`uname -r`.debug
To find one with a matching build id (.notes ELF section, then
nhdr->n_type == NT_GNU_BUILD_ID), just like the Linux kernel 'perf
tools', where this code comes from, with some minor modifications to
cope with not having symbol_conf, symfs, etc.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Take 'struct task_struct' in the Linux kernel, these fields:
/* --- cacheline 2 boundary (128 bytes) --- */
struct sched_entity se; /* 128 448 */
/* XXX last struct has 24 bytes of padding */
/* --- cacheline 9 boundary (576 bytes) --- */
struct sched_rt_entity rt; /* 576 48 */
The sched_entity struct has 24 bytes of padding, and that info would
only appear when printing 'struct task_struct' if class__find_holes()
had previously been run on 'struct sched_entity' which wasn't always the
case, make sure that happens.
This results in this extra stat being printed for 'struct task_struct':
/* paddings: 4, sum paddings: 38 */
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Need to read more on http://www.artima.com/cppsource/rvalue.html, but
handling it mostly like DW_TAG_typedef so that at least references to it
are resolved, we can get its byte size, etc.
FIXME: look at the vtable parameters, some are resolving to "(null)".
Reported-by: Benjamin Kosnik <bkoz@redhat.com>
Reported-by: Mark Wieelard <mjw@redhat.com>
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=962571
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Silly bug, it was trying to return a negative index stating what file
had problems in the provided array, but if the first had problems it was
return -0, duh, fix it by returning the first as 1, etc.
With this, calling 'pdwtags non-existent-file' would return no errors
via $?.
Next csets will provide proper error messages, using what is in errno
and this index to tell what file has problems.
Reported-by: Eric Blake <eblake@redhat.com>
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=949034
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
As Thomas Gleixner wisely pointed out, using 'self' is stupid, it
doesn't convey useful information, so use sensible names.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Needed because it uses S_ISDIR. In the past this header probably was
being indirectly included. Noticed while building on RHEL6 Beta.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
[acme@doppio pahole]$ pahole -F ctf /media/tb/debuginfo/usr/lib/debug/usr/bin/greycstoration4integration.debug > /tmp/bla
<ERROR(tag__size:837): detected type loop: type=572, tag=const_type>
<ERROR(tag__size:837): detected type loop: type=572, tag=const_type>
[acme@doppio pahole]$
These type loops are problems in the CTF encoding, that should be fixed, but
should not cause the core code to segfault on an infinite recursion.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
That asks dwarf_fprintf to always use "struct" in places where it would
use "class", because CTF doesn't have the "class" concept, so for
'regtest diffctf' sake, we use this.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
It doesn't matter when using a traditional malloc/free allocator, but
with obstacks we need to do it in reverse order.
For the usual case where we successfully process an object this doesn't
matter, as when we started using obstacks we don't traverse all the tags
calling their destructors anymore, we just free the whole obstack in one
go.
Noticed when processing object files built from non-supported languages
such as FORTRAN and Pascal, where there are some DWARF tags that are not
supported, which makes the object file load to be prematurely aborted
and that calls destructors for things like classes and functions that in
turn free space for their parameter/member lists, which now have to be
done in reverse order.
We could just stop calling the destructors and then destroying the whole
obstack, but I think that partially processed files are a nice feature,
so keep the interface in a way that both obstacks and traditinal malloc
alocators can be used.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Next we'll add a new kind of tag, DW_TAG_perf_counter, that will come
from perf.data generated by 'perf report'.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
That is used by cus__find_function_by_addr & cu__func_function_by_addr.
First user is pfunct --addr, but this is really for pfunct --annotate, that
will process a perf.data file generated by 'perf report', load the debugging
info and regenerate the functions (pfunct -TVi like) that had hits, using
libdisasm to show the assembly code, etc.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Andrii Nakryiko
Arnaldo Carvalho de Melo
Domenico Andreoli
Yonghong Song