Existing code base assumes that single CU doesn't have more than 65535
types per each CU, which might be a reasonable assumption for DWARF
data. With BTF, though, all we get is single, potentially huge, CU which
can easily have more than 65k types. For example, this is the case for
allyesconfig version of Linux kernel, which has >200k types.
Due to this assumption, libdwarves and other parts of pahole are using
16-bit counters to iterate over entities within CU. This can cause
infinite loops when iterating BTF data, if there are more than 65535
types. This patch changes non-public variables to use 32-bit integers,
where appropriate.
This still leads to invalid reported data when using BTF loader (due to using
(X & 0xFFFF) type ID, instead of X, when X > 65535) and loading huge files,
but at least it's not stuck in an infinite loop anymore.
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Reported-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Alexei Starovoitov <ast@fb.com>
Cc: Yonghong Song <yhs@fb.com>
Cc: bpf@vger.kernel.org
Cc: dwarves@vger.kernel.org
[ Removed non type ID conversions, for instance for the kind of tag, like in type->namespace.tag.tag, that can remain a uint16_t ]
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Now unions are handled as well in pahole, so bail out before checking
'struct class' members for struct specific filters when handling
non-structs in class__filter()
Tested-by: Andrii Nakryiko <andriin@fb.com>
Fixes: 31664d60ad ("pahole: Show tagged enums as well when no class is specified")
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
This patch fixed two issues with BTF. One is related to struct/union
bitfield encoding and the other is related to forward type.
Issue #1 and solution:
======================
Current btf encoding of bitfield follows what pahole generates.
For each bitfield, pahole will duplicate the type chain and
put the bitfield size at the final int or enum type.
Since the BTF enum type cannot encode bit size,
commit b18354f64c ("btf: Generate correct struct bitfield
member types") workarounds the issue by generating
an int type whenever the enum bit size is not 32.
The above workaround is not ideal as we lost original type
in BTF. Another undesiable fact is the type duplication
as the pahole duplicates the type chain.
To fix this issue, this patch implemented a compatible
change for BTF struct type encoding:
. the bit 31 of type->info, previously reserved,
now is used to indicate whether bitfield_size is
encoded in btf_member or not.
. if bit 31 of struct_type->info is set,
btf_member->offset will encode like:
bit 0 - 23: bit offset
bit 24 - 31: bitfield size
if bit 31 is not set, the old behavior is preserved:
bit 0 - 31: bit offset
So if the struct contains a bit field, the maximum bit offset
will be reduced to (2^24 - 1) instead of MAX_UINT. The maximum
bitfield size will be 255 which is enough for today as maximum
bitfield in compiler can be 128 where int128 type is supported.
A new global, no_bitfield_type_recode, is introduced and which
will be set to true if BTF encoding is enabled. This global
will prevent pahole duplicating the bitfield types to avoid
type duplication in BTF.
Issue #2 and solution:
======================
Current forward type in BTF does not specify whether the original
type is struct or union. This will not work for type pretty print
and BTF-to-header-file conversion as struct/union must be specified.
To fix this issue, similar to issue #1, type->info bit 31
is used. If the bit is set, it is union type. Otherwise, it is
a struct type.
Examples:
=========
-bash-4.4$ cat t.c
struct s;
union u;
typedef int ___int;
enum A { A1, A2, A3 };
struct t {
int a[5];
___int b:4;
volatile enum A c:4;
struct s *p1;
union u *p2;
} g;
-bash-4.4$ gcc -c -O2 -g t.c
Without this patch:
$ pahole -JV t.o
[1] TYPEDEF ___int type_id=2
[2] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
[3] ENUM A size=4 vlen=3
A1 val=0
A2 val=1
A3 val=2
[4] STRUCT t size=40 vlen=5
a type_id=5 bits_offset=0
b type_id=13 bits_offset=160
c type_id=15 bits_offset=164
p1 type_id=9 bits_offset=192
p2 type_id=11 bits_offset=256
[5] ARRAY (anon) type_id=2 index_type_id=2 nr_elems=5
[6] INT sizetype size=8 bit_offset=0 nr_bits=64 encoding=(none)
[7] VOLATILE (anon) type_id=3
[8] FWD s type_id=0
[9] PTR (anon) type_id=8
[10] FWD u type_id=0
[11] PTR (anon) type_id=10
[12] INT int size=1 bit_offset=0 nr_bits=4 encoding=(none)
[13] TYPEDEF ___int type_id=12
[14] INT (anon) size=1 bit_offset=0 nr_bits=4 encoding=SIGNED
[15] VOLATILE (anon) type_id=14
With this patch:
$ pahole -JV t.o
File t.o:
[1] TYPEDEF ___int type_id=2
[2] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
[3] ENUM A size=4 vlen=3
A1 val=0
A2 val=1
A3 val=2
[4] STRUCT t kind_flag=1 size=40 vlen=5
a type_id=5 bitfield_size=0 bits_offset=0
b type_id=1 bitfield_size=4 bits_offset=160
c type_id=7 bitfield_size=4 bits_offset=164
p1 type_id=9 bitfield_size=0 bits_offset=192
p2 type_id=11 bitfield_size=0 bits_offset=256
[5] ARRAY (anon) type_id=2 index_type_id=2 nr_elems=5
[6] INT sizetype size=8 bit_offset=0 nr_bits=64 encoding=(none)
[7] VOLATILE (anon) type_id=3
[8] FWD s struct
[9] PTR (anon) type_id=8
[10] FWD u union
[11] PTR (anon) type_id=10
The fix removed the type duplication, preserved the enum type for the
bitfield, and have correct struct/union information for the forward
type.
Signed-off-by: Yonghong Song <yhs@fb.com>
Acked-by: Martin KaFai Lau <kafai@fb.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Alexei Starovoitov <ast@fb.com>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Currently, the pahole dwarf->btf conversion only supports one
compilation unit. This is not ideal since we would like using pahole to
generate BTF for vmlinux which has a lot of compilation units.
This patch added support to process multiple compilation units per ELF
file. Multiple ELF files are also supported properly.
The following is a demonstration example:
-bash-4.4$ cat t1.c
struct t1 {
int a1;
} g1;
int main(void) { return 0; }
-bash-4.4$ cat t2.c
struct t2 {
char a2;
} g2;
int main() { return 0; }
-bash-4.4$ cat t3.c
struct t3 {
unsigned char a1:4;
} g1;
int main(void) { return 0; }
-bash-4.4$ cat t4.c
struct t4 {
volatile char a4;
} g2;
int main() { return 0; }
-bash-4.4$ gcc -O2 -o t1 -g t1.c t2.c
-bash-4.4$ gcc -O2 -o t3 -g t3.c t4.c
Note that both the binary "t1" and "t3" have two compilation units in
their respective dwarf debug_info sections. The following is the pahole
verbose output for BTF conversion for these two binaries.
-bash-4.4$ pahole -JV t1 t3
File t1:
[1] STRUCT t1 size=4 vlen=1
a1 type_id=2 bits_offset=0
[2] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
[3] STRUCT t2 size=1 vlen=1
a2 type_id=4 bits_offset=0
[4] INT char size=1 bit_offset=0 nr_bits=8 encoding=(none)
[5] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
File t3:
[1] STRUCT t3 size=1 vlen=1
a1 type_id=3 bits_offset=0
[2] INT unsigned char size=1 bit_offset=0 nr_bits=8 encoding=(none)
[3] INT unsigned char size=1 bit_offset=0 nr_bits=4 encoding=(none)
[4] INT (anon) size=4 bit_offset=0 nr_bits=32 encoding=(none)
[5] STRUCT t4 size=1 vlen=1
a4 type_id=6 bits_offset=0
[6] VOLATILE (anon) type_id=7
[7] INT char size=1 bit_offset=0 nr_bits=8 encoding=(none)
[8] INT int size=4 bit_offset=0 nr_bits=32 encoding=SIGNED
Signed-off-by: Andrii Nakryiko <andriin@fb.com>
Acked-by: Martin KaFai Lau <kafai@fb.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Alexei Starovoitov <ast@fb.com>
Cc: Yonghong Song <yhs@fb.com>
Signed-off-by: Yonghong Song <yhs@fb.com>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
The pahole tool initial goal was to show struct holes, which can't
happen with a first level of a tagged union (a union with a name),
so those only were displayed when part of a higher level struct.
Show the first level tagged enums as well, as this is useful when just
wanting to see its members.
E.g.:
$ pahole ../build/v4.20-rc5/net/core/sock.o | grep ^union
union fpregs_state {
union irq_stack_union {
union sigval {
union __sifields {
union thread_union {
union kernfs_node_id {
union flowi_uli {
union ethtool_flow_union {
union key_payload {
union bpf_attr {
union tcp_md5_addr {
$
Suggested-by: Matthew Wilcox <willy@infradead.org>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
This patch introduces BPF Type Format (BTF).
BTF (BPF Type Format) is the meta data format which describes
the data types of BPF program/map. Hence, it basically focus
on the C programming language which the modern BPF is primary
using. The first use case is to provide a generic pretty print
capability for a BPF map.
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Seems like a nice shortcut for kernel hackers, and one that makes sure
the right vmlinux file is used, as it reads the running kernel build id
from /sys/kernel/notes and then searches the well known path for vmlinux
files:
vmlinux
/boot/vmlinux
/boot/vmlinux-4.14.0+
/usr/lib/debug/boot/vmlinux-`uname -r`
/lib/modules/`uname -r`/build/vmlinux
/usr/lib/debug/lib/modules/`uname -r`/vmlinux
/usr/lib/debug/boot/vmlinux-`uname -r`.debug
To find one with a matching build id (.notes ELF section, then
nhdr->n_type == NT_GNU_BUILD_ID), just like the Linux kernel 'perf
tools', where this code comes from, with some minor modifications to
cope with not having symbol_conf, symfs, etc.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
$ cat foo
cat: foo: No such file or directory
Before:
$ pahole foo
pahole: No debugging information found
After:
$ pahole foo
pahole: foo: No such file or directory
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
As Thomas Gleixner wisely pointed out, using 'self' is stupid, it
doesn't convey useful information, so use sensible names.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
.debug_types is a new DWARF 4 feature that adds some simple
compression to DWARF. The basic idea is that identical type
definitions are merged by the linker and put into a new .debug_types
section.
This introduces 'dwarf_off_ref', which is like Dwarf_Off, but carries
a flag indicating if the DIE came from .debug_types. This allows
future uses to find the DIE properly.
Type units are read a little differently from ordinary CUs. All type
units are read at once, then types are recoded for them. (I think
something like this is needed more generally, to support inter-CU
references, but I have not tried that.)
The type unit is also kept around longer, because any other CU may
refer to it. This necessitated a change to load_steal_kind to replace
the notion of a "stolen" CU with a request by the "stealer" for the
caller to delete the CU when appropriate.
I need elfutils patch 581c3f60e2b1fc7ddaf4260bb5a9cb59f8e3f0d0
to make this work properly; without it I see crashes.
You can make test cases by compiling with '-gdwarf-4 -fdebug-types-section'.
As it is corrupting the obstack and it will be deleted completely at
cu__delete time anyway. Stick a FIXME tho, as it is good to completely
understand and fix this eventually.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
That asks dwarf_fprintf to always use "struct" in places where it would
use "class", because CTF doesn't have the "class" concept, so for
'regtest diffctf' sake, we use this.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
For now --encode_ctf/-Z only encodes the tags in the first object file
(CU) in a multi-obj/CU file, so for regtest sake, for now, introduce
this option.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Where we end up with the same struct but class__size() returns off by
some bytes result. Investigating...
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Only when debugging leaks we need to be so judicious.
When not debugging we want to exit faster :-)
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
We really only need structures__add, that now returns an existing entry
or add a new one, the code that uses it just passes a bool to figure out
if this is a new entry or an existing one.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Replicating the comment added to the source code:
No sense in adding an anonymous struct to the list of structs already
printed, as we look for the name... The right fix probably will be to
call class__fprintf on a in-memory FILE, do a hash, and look it by full
contents, not by name. And this is needed for CTF as well, but its late
now and I'm sleepy, will leave for later...
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
I wasn't especifying the optimization level and the default, despite
using -Wall, was for this so simple case not to be warned about, so
now I'm using -O2.
Alexandre provided a patch initializing the variables to NULL, so that
when we called cus__delete it would bail out and not possibly act on
a random value, I preferred to add extra goto labels and do the exit
path only on the resources that were successfully allocated/initialized,
avoiding, for instance, to call dwarves_exit() if dwarves_init() wasn't
called, which wasn't a problem so far, but could be in the future.
Reported-by: Alexandre Vassalotti <alexandre@peadrop.com>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Instead pass thru cu__strings(cu, i) so that we can figure out if the
underlying debugging format handler can do that more efficiently, such as by
looking up directly the ELF section ".strtab".
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
It ain't gonna be like that soon, when every cu that comes from a CTF file will
have all its strings_t pointing to strings in .strtab.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Too DWARF specific and wasn't working since I implemented type recoding and
removed the Dwarf_Off from struct tag.
To achieve the same result one can use --show_decl_info that also shows the
dwarf offset, since it needs to keep the DWARF specific line number and file
and then use a text editor to find the offset.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
As we will show them on the first level. This is yet another thing
required to properly compare te result of "pahole -F ctf foo" with the
output of "pahole -F dwarf foo", as there is no support, that I know of,
for namespacing in ctf.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
By default pahole doesn't prints structs/classes that are only defined
inside functions, so add a knob to aks for that.
This is for the benefit of ctfdwdiff, as in CTF we don't have
expressiveness to tell that a struct is only defined inside a function,
its all in the global table.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
CTF doesn't have support for multiple array dimensions, so it flattens
the arrays.
This caused a large number of false positives in ctfdwdiff, so introduce
this conf_fprintf option, use it in pahole and ctfdwdiff.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
For a file with just DWARF info:
$ pahole -F ctf build/pahole
$
But if we ask that it also try dwarf:
$ pahole -F ctf,dwarf build/pahole | head -2
struct _IO_FILE {
int _flags; /* 0 4 */
$
Useful when testing the new CTF support in these tools, as we'll be able to,
from the DWARF info in objects, generate the CTF equivalent and add to the same
object, then run pahole -A -F ctf, pahole -A -F dwarf and compare the outputs.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
"pahole -Z foo" will create foo.SUNW_ctf, that if objcopy
--add-section'ed to the right word-sized object will work, sans VARARGS,
that will get fixed soon (as in, probably, tomorrow).
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
To shorten the name and to reflect the fact that we're no longer
"finding" a type, but merely accessing an array with a bounds check in
this function.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Amazing how many crept up over time, should have set the
execute bit of .git/hooks/pre-commit already, duh.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
So that we can, for instance, go from:
[acme@doppio pahole]$ time pahole -C sk_buff
/usr/lib/debug/lib/modules/2.6.27.19-170.2.35.fc10.x86_64/vmlinux | head
-3
struct sk_buff {
struct sk_buff * next; /* 0 8 */
struct sk_buff * prev; /* 8 8 */
real 0m11.071s
user 0m10.627s
sys 0m0.359s
[acme@doppio pahole]$
To a mere:
[acme@doppio pahole]$ time pahole -C sk_buff
/usr/lib/debug/lib/modules/2.6.27.19-170.2.35.fc10.x86_64/vmlinux | head
-3
struct sk_buff {
struct sk_buff * next; /* 0 8 */
struct sk_buff * prev; /* 8 8 */
real 0m1.464s
user 0m1.431s
sys 0m0.016s
[acme@doppio pahole]$
And also results go appearing much quicker for other options, etc.
The other tools will be converted too, but the old way of working will
always be there, as it is possible that we may need all the types in
memory for some future tool.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Also introducing cus__load, that load just one file.
The new cus__load_files routine now iterates thru the provided array
calling cus__load for each, and that in turn will try first dwarf__load,
and if that fail, i.e. if no DWARF info is found, call ctf__load.
This now allows loading DWARF _and_ CTF files at the same time. This
will be useful in the future when we, from DWARF generate CTF and at the
same time do a codiff, comparing the freshly generated CTF file with the
DWARF it came from.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
So that, when not needing the DWARF info, the apps can tell that at load
time, and then the dwarf loader can just free all the dwarf_tags
allocated, reducing memory usage.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
To take advantage of cache effects and to avoid calling cu__find_holes
more than once on the same struct.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Andrii Nakryiko
Domenico Andreoli
Arnaldo Carvalho de Melo
Yonghong Song
Martin KaFai Lau
Tom Tromey