To ask for sorting output, initially by name.
This is needed in 'btfdiff' to diff the output of 'pahole -F dwarf
--jobs N', where N threads will go on consuming DWARF compile units and
and pretty printing them, producing a non deterministic output.
So we need to sort the output for both BTF and DWARF, and then diff
them.
This is still not enough for some cases where different types have the
same name, things like "usb_priv" that exists in multiple DWARF compile
units, the first processed is "winning", i.e. being the only one
considered.
I have to look how BTF handles this to adopt a similar algorithm and
keep btfdiff usable as a regression test for the BTF and DWARF loader
and the BTF encoder.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Using the same command line option as 'make' and 'ninja': -j.
Unfortunately the argp parser in glibc expects a '=' for single letter
args, so it is:
$ pahole -j=10
or, like with 'make':
$ make --jobs=10
This is unwired at the moment, probably I'll reorder the patch, but for
testing, add it first.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
The '-j' option, to generate detached BTF files wasn't published in any
tagged release, so leave it as just --btf_encode_detached and make '-j'
available for specifying the number of jobs/threads, as is the case with
'make -j', 'ninja -j', etc.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
To solve problems similar to _RH_KABI_REPLACE. The _RH_KABI_REPLACE(_orig, _new) macros perserve size alignment and kabi agreement between _orig and _new.Below is the definition of this macro:
union { \
_new; \
struct { \
_orig; \
} __UNIQUE_ID(rh_kabi_hide); \
__RH_KABI_CHECK_SIZE_ALIGN(_orig, _new); \
}
__UNIQUE_ID uses the __COUNTER__ macro, and the __COUNTER__ macro is automatically incremented by 1 every time it is precompiled. Therefore, in different compilation units, the same structure has different names.Here is a concrete example:
struct acpi_dev_node {
union {
struct acpi_device *companion;
struct {
void *handle;
} __UNIQUE_ID_rh_kabi_hide29;
union { };
};
};
struct acpi_dev_node {
union {
struct acpi_device *companion;
struct {
void *handle;
} __UNIQUE_ID_rh_kabi_hide31;
union { };
};
};
Finally, it will cause the btf algorithm to de-duplication efficiency is not high, and time-consuming. For example, running ./pahole -J vmlinux-3.10.0-1160.el7.x86_64 without --kabi_prefix flag, the running time is:
real 8m28.912s
user 8m27.271s
sys 0m1.471s
And the size of the generated btf segment is 30678240 bytes.
After adding the patch, running ./pahole --kabi_prefix=__UNIQUE_ID_rh_kabi_hide -J vmlinux-3.10.0-1160.el7.x86_64. The running time of the command is:
real 0m19.634s
user 0m18.457s
sys 0m1.169s
And the size of the generated btf segment is 3117719 bytes.
Signed-off-by: Shuyi Cheng <chengshuyi@linux.alibaba.com>
Acked-by: Jiri Olsa <jolsa@redhat.com>
Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Wenan Mao <wenan.mao@linux.alibaba.com>
Cc: dwarves@vger.kernel.org
Link: https://lore.kernel.org/dwarves/482e5543-d7da-7bed-098d-cc879d8db253@linux.alibaba.com/
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
By default, pahole makes use only of BTF features introduced with kernel
v5.2. Features that are added later need to be turned on with explicit
feature flags, such as --btf_gen_floats. According to [1], this will
hinder the people who generate BTF for kernels externally (e.g. for old
kernels to support BPF CO-RE).
Introduce --btf_gen_all that allows using all BTF features supported
by pahole.
[1] https://lore.kernel.org/dwarves/CAEf4Bzbyugfb2RkBkRuxNGKwSk40Tbq4zAvhQT8W=fVMYWuaxA@mail.gmail.com/
Signed-off-by: Ilya Leoshkevich <iii@linux.ibm.com>
Suggested-by: Andrii Nakryiko <andrii@kernel.org>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Yonghong Song <yhs@fb.com>
Cc: bpf@vger.kernel.org
Cc: dwarves@vger.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Some BPF programs compiled on s390 fail to load, because s390
arch-specific linux headers contain float and double types.
Fix as follows:
- Make the DWARF loader fill base_type.float_type.
- Introduce the --btf_gen_floats command-line parameter, so that
pahole could be used to build both the older and the newer kernels.
- libbpf introduced the support for the floating-point types in commit
986962fade5, so update the libbpf submodule to that version and use
the new btf__add_float() function in order to emit the floating-point
types when not in the compatibility mode.
- Make the BTF loader recognize the new BTF kind.
Example of the resulting entry in the vmlinux BTF:
[7164] FLOAT 'double' size=8
when building with:
LLVM_OBJCOPY=${OBJCOPY} ${PAHOLE} -J ${1} --btf_gen_floats
Signed-off-by: Ilya Leoshkevich <iii@linux.ibm.com>
Acked-by: Andrii Nakryiko <andrii@kernel.org>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Cc: Heiko Carstens <hca@linux.ibm.com>
Cc: Vasily Gorbik <gor@linux.ibm.com>
Cc: Yonghong Song <yhs@fb.com>
Cc: bpf@vger.kernel.org
Cc: dwarves@vger.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Add support for generating split BTF, in which there is a designated base
BTF, containing a base set of types, and a split BTF, which extends main BTF
with extra types, that can reference types and strings from the main BTF.
This is going to be used to generate compact BTFs for kernel modules, with
vmlinux BTF being a main BTF, which all kernel modules are based off of.
These changes rely on patch set [0] to be present in libbpf submodule.
[0] https://patchwork.kernel.org/project/netdevbpf/list/?series=377859&state=*
Committer notes:
Fixed up wrt ARGP_numeric_version and added a man page entry.
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Cc: bpf@vger.kernel.org
Cc: dwarves@vger.kernel.org
Cc: kernel-team@fb.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
In Makefiles we want to do purely numeric comparisions, such as in the
Linux kernel Makefiles and scripts, that have things like this at the
moment:
$ grep PAHOLE */*.sh
scripts/link-vmlinux.sh: if ! [ -x "$(command -v ${PAHOLE})" ]; then
scripts/link-vmlinux.sh: echo >&2 "BTF: ${1}: pahole (${PAHOLE}) is not available"
scripts/link-vmlinux.sh: pahole_ver=$(${PAHOLE} --version | sed -E 's/v([0-9]+)\.([0-9]+)/\1\2/')
scripts/link-vmlinux.sh: echo >&2 "BTF: ${1}: pahole version $(${PAHOLE} --version) is too old, need at least v1.16"
scripts/link-vmlinux.sh: LLVM_OBJCOPY=${OBJCOPY} ${PAHOLE} -J ${1}
$
So just provide:
$ pahole --numeric_version
118
$
While keeping the --version output for older Makefiles and scripts.
Cc: Andrii Nakryiko <andrii@kernel.org>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
To show just packed structs.
For instance, here are the top packed structures in the Linux kernel,
using BTF data:
$ pahole --packed --sizes | sort -k2 -nr | head
e820_table 64004 0
boot_params 4096 0
btrfs_super_block 3531 0
efi_variable 2084 0
ntb_info_regs 800 0
tboot 568 0
_legacy_mbr 512 0
disklabel 512 0
btrfs_root_item 439 0
saved_context 317 0
$
If you then look at:
$ pahole e820_table
struct e820_table {
__u32 nr_entries; /* 0 4 */
struct e820_entry entries[3200]; /* 4 64000 */
/* size: 64004, cachelines: 1001, members: 2 */
/* last cacheline: 4 bytes */
} __attribute__((__packed__));
$
In arch/x86/include/asm/e820/types.h we have:
/*
* The whole array of E820 entries:
*/
struct e820_table {
__u32 nr_entries;
struct e820_entry entries[E820_MAX_ENTRIES];
};
I.e. no explicit __packed__ attributes, but if we expand this a bit:
$ pahole -E e820_table
struct e820_table {
/* typedef __u32 */ unsigned int nr_entries; /* 0 4 */
struct e820_entry {
/* typedef u64 -> __u64 */ long long unsigned int addr; /* 4 8 */
/* typedef u64 -> __u64 */ long long unsigned int size; /* 12 8 */
enum e820_type type; /* 20 4 */
} __attribute__((__packed__)) entries[3200]; /* 4 64000 */
/* size: 64004, cachelines: 1001, members: 2 */
/* last cacheline: 4 bytes */
} __attribute__((__packed__));
$
We see that is that entries member that is packed, because:
$ pahole e820_entry
struct e820_entry {
u64 addr; /* 0 8 */
u64 size; /* 8 8 */
enum e820_type type; /* 16 4 */
/* size: 20, cachelines: 1, members: 3 */
/* last cacheline: 20 bytes */
} __attribute__((__packed__));
$
In arch/x86/include/asm/e820/types.h we have:
/*
* A single E820 map entry, describing a memory range of [addr...addr+size-1],
* of 'type' memory type:
*
* (We pack it because there can be thousands of them on large systems.)
*/
struct e820_entry {
u64 addr;
u64 size;
enum e820_type type;
} __attribute__((packed));
So yeah, it is there, BTF doesn't explicitly states it is packed (as
DWARF does) and pahole was able to infer that correctly.
Tested-by: Richard Weinberger <richard@nod.at>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Commit f3d9054ba8 ("btf_encoder: Teach pahole to store percpu
variables in vmlinux BTF.") introduced an option '-j' that makes
effort in emitting VAR entries in BTF. Before no one has been using
this flag, replace the one-letter option '-j' with a full flag name
'--btf_encode_force' to save '-j' for future uses.
Committer notes:
Added missing man page entry.
Signed-off-by: Hao Luo <haoluo@google.com>
Cc: Alexei Starovoitov <alexei.starovoitov@gmail.com>
Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Cc: Yonghong Song <yhs@fb.com>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
A new feature was introduced in commit f3d9054ba8 ("btf_encoder: Teach
pahole to store percpu variables in vmlinux BTF.") which encodes kernel
percpu variables into BTF. Add a flag --skip_encoding_btf_vars to allow
users to toggle this feature off, so that the rollout of pahole v1.18
can be protected by potential bugs in this feature.
Committer notes:
Added missing man page entry.
Signed-off-by: Hao Luo <haoluo@google.com>
Cc: Alexei Starovoitov <alexei.starovoitov@gmail.com>
Cc: Andrii Nakryiko <andrii.nakryiko@gmail.com>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Cc: Yonghong Song <yhs@fb.com>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
See the pahole man page:
Some of the records are not found in 'type_enum=perf_event_type' so some of the
records don't get converted to a type that fully shows its contents. For perf
we know that those are in another enumeration, 'enum perf_user_event_type', so,
for these cases, we can create a 'virtual enum', i.e. the sum of two enums
and then get all those entries decoded and properly casted, first few records
with just 'enum perf_event_type':
$ pahole ~/bin/perf --header=perf_file_header --seek_bytes '$header.data.offset' --size_bytes='$header.data.size' -C 'perf_event_header(sizeof,type,type_enum=perf_event_type)' --count 4 < perf.data
{
.type = 79,
.misc = 0,
.size = 32,
},
{
.type = 73,
.misc = 0,
.size = 40,
},
{
.type = 74,
.misc = 0,
.size = 32,
},
{
.header = {
.type = PERF_RECORD_CGROUP,
.misc = 0,
.size = 40,
},
.id = 1,
.path = "/",
},
$
Now with both enumerations, i.e. with 'type_enum=perf_event_type+perf_user_event_type':
$ pahole ~/bin/perf --header=perf_file_header --seek_bytes '$header.data.offset' --size_bytes='$header.data.size' -C 'perf_event_header(sizeof,type,type_enum=perf_event_type+perf_user_event_type)' --count 5 < perf.data
{
.header = {
.type = PERF_RECORD_TIME_CONV,
.misc = 0,
.size = 32,
},
.time_shift = 31,
.time_mult = 1016803377,
.time_zero = 435759009518382,
},
{
.header = {
.type = PERF_RECORD_THREAD_MAP,
.misc = 0,
.size = 40,
},
.nr = 1,
.entries = 0x50 0x7e 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00,
},
{
.header = {
.type = PERF_RECORD_CPU_MAP,
.misc = 0,
.size = 32,
},
.data = {
.type = 1,
.data = "",
},
},
{
.header = {
.type = PERF_RECORD_CGROUP,
.misc = 0,
.size = 40,
},
.id = 1,
.path = "/",
},
{
.header = {
.type = PERF_RECORD_CGROUP,
.misc = 0,
.size = 48,
},
.id = 1553,
.path = "/system.slice",
},
$
And since the fun never ends, this needs to be properly supported (arrays of
structs):
$ pahole ~/bin/perf -C perf_record_thread_map_entry
struct perf_record_thread_map_entry {
__u64 pid; /* 0 8 */
char comm[16]; /* 8 16 */
/* size: 24, cachelines: 1, members: 2 */
/* last cacheline: 24 bytes */
};
$
that 'nr' field:
$ pahole ~/bin/perf -C perf_record_thread_map
struct perf_record_thread_map {
struct perf_event_header header; /* 0 8 */
__u64 nr; /* 8 8 */
struct perf_record_thread_map_entry entries[]; /* 16 0 */
/* size: 16, cachelines: 1, members: 3 */
/* last cacheline: 16 bytes */
};
$
So probably we need something like a file with types and its pretty printing
details, with one for 'struct perf_record_thread_map':
perf_record_thread_map(entries.nr_entries=$nr)
Meaning: the perf_record_thread_map 'entries' has a number of
sizeof(typeof entries) that is defined in the perf_record_thread_map
'nr' member. Everything starting with a '$' needs to be evaluated, like
the '$header.*' we already have, since there is no 'namespace selector'
($header. in the header case) this means: the current record.
So, when pretty printing a record that has such type (perf_record_thread_map)
this has to be taken into account, and validated by the containing 'struct
perf_event_header->size' field.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
I.e. these are all equivalent:
filter=type==PERF_RECORD_MMAP
type==PERF_RECORD_MMAP
filter=type==MMAP
type==MMAP
Update docs about it.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
The default is to show structs and unions, if just structs should be
considered, use --structs, just don't use it together with --unions or
nothing will be shown 8-)
$ pahole --find_pointers_to ehci_qh
ehci_hcd: qh_scan_next
ehci_hcd: async
ehci_hcd: dummy
ehci_shadow: qh
$ pahole --structs --find_pointers_to ehci_qh
ehci_hcd: qh_scan_next
ehci_hcd: async
ehci_hcd: dummy
$ pahole --unions --find_pointers_to ehci_qh
ehci_shadow: qh
$ pahole --structs --unions --find_pointers_to ehci_qh
$
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
We use things like DW_AT_alignment, so not all of those attributes are
inferred by formats like BTF that lack that info, allow suppressing the
output and make btfdiff ask for both DWARF and BTF output to have this
suppressed.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
So that we can use it in things like btfdiff.
Cc: Alexei Starovoitov <ast@fb.com>
Cc: Andrii Nakryiko <andriin@fb.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Mark Wielaard <mark@klomp.org>
Cc: Yonghong Song <yhs@fb.com>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Flavio needed this and found it only after some head scratching, add it.
Someone could help here by looking at other missing entries in the man
page, getting it in synch with 'pahole --help' ;-)
Reported-by: Flavio Leitner <fbl@redhat.com>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
The ghostprotocols.net domain is busted for a while, point to the
more permanent OLS proceedings pdf hosted at kernel.org.
Reported-by: Borislav Petkov <bp@alien8.de>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
CTF doesn't have support for multiple array dimensions, so it flattens
the arrays.
This caused a large number of false positives in ctfdwdiff, so introduce
this conf_fprintf option, use it in pahole and ctfdwdiff.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
For a file with just DWARF info:
$ pahole -F ctf build/pahole
$
But if we ask that it also try dwarf:
$ pahole -F ctf,dwarf build/pahole | head -2
struct _IO_FILE {
int _flags; /* 0 4 */
$
Useful when testing the new CTF support in these tools, as we'll be able to,
from the DWARF info in objects, generate the CTF equivalent and add to the same
object, then run pahole -A -F ctf, pahole -A -F dwarf and compare the outputs.
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Arnaldo Carvalho de Melo
Shuyi Cheng
Ilya Leoshkevich
Andrii Nakryiko
Hao Luo