Python-lang gdb script to extract x86_64 guest vmcore from qemu coredump
When qemu dies unexpectedly, for example in response to an explicit
abort() call, or (more importantly) when an external signal is delivered
to it that results in a coredump, sometimes it is useful to extract the
guest vmcore from the qemu process' memory image. The guest vmcore might
help understand an emulation problem in qemu, or help debug the guest.
This script reimplements (and cuts many features of) the
qmp_dump_guest_memory() command in gdb/Python,
https://sourceware.org/gdb/current/onlinedocs/gdb/Python-API.html
working off the saved memory image of the qemu process. The docstring in
the patch (serving as gdb help text) describes the limitations relative to
the QMP command.
Dependencies of qmp_dump_guest_memory() have been reimplemented as needed.
I sought to follow the general structure, sticking to original function
names where possible. However, keeping it simple prevailed in some places.
The patch has been tested with a 4 VCPU, 768 MB, RHEL-6.4
(2.6.32-358.el6.x86_64) guest:
- The script printed
> guest RAM blocks:
> target_start target_end host_addr message count
> ---------------- ---------------- ---------------- ------- -----
> 0000000000000000 00000000000a0000 00007f95d0000000 added 1
> 00000000000a0000 00000000000b0000 00007f960ac00000 added 2
> 00000000000c0000 00000000000ca000 00007f95d00c0000 added 3
> 00000000000ca000 00000000000cd000 00007f95d00ca000 joined 3
> 00000000000cd000 00000000000d0000 00007f95d00cd000 joined 3
> 00000000000d0000 00000000000f0000 00007f95d00d0000 joined 3
> 00000000000f0000 0000000000100000 00007f95d00f0000 joined 3
> 0000000000100000 0000000030000000 00007f95d0100000 joined 3
> 00000000fc000000 00000000fc800000 00007f960ac00000 added 4
> 00000000fffe0000 0000000100000000 00007f9618800000 added 5
> dumping range at 00007f95d0000000 for length 00000000000a0000
> dumping range at 00007f960ac00000 for length 0000000000010000
> dumping range at 00007f95d00c0000 for length 000000002ff40000
> dumping range at 00007f960ac00000 for length 0000000000800000
> dumping range at 00007f9618800000 for length 0000000000020000
- The vmcore was checked with "readelf", comparing the results against a
vmcore written by qmp_dump_guest_memory():
> --- theirs 2013-09-12 17:38:59.797289404 +0200
> +++ mine 2013-09-12 17:39:03.820289404 +0200
> @@ -27,16 +27,16 @@
> Type Offset VirtAddr PhysAddr
> FileSiz MemSiz Flags Align
> NOTE 0x0000000000000190 0x0000000000000000 0x0000000000000000
> - 0x0000000000000ca0 0x0000000000000ca0 0
> - LOAD 0x0000000000000e30 0x0000000000000000 0x0000000000000000
> + 0x000000000000001c 0x000000000000001c 0
> + LOAD 0x00000000000001ac 0x0000000000000000 0x0000000000000000
> 0x00000000000a0000 0x00000000000a0000 0
> - LOAD 0x00000000000a0e30 0x0000000000000000 0x00000000000a0000
> + LOAD 0x00000000000a01ac 0x0000000000000000 0x00000000000a0000
> 0x0000000000010000 0x0000000000010000 0
> - LOAD 0x00000000000b0e30 0x0000000000000000 0x00000000000c0000
> + LOAD 0x00000000000b01ac 0x0000000000000000 0x00000000000c0000
> 0x000000002ff40000 0x000000002ff40000 0
> - LOAD 0x000000002fff0e30 0x0000000000000000 0x00000000fc000000
> + LOAD 0x000000002fff01ac 0x0000000000000000 0x00000000fc000000
> 0x0000000000800000 0x0000000000800000 0
> - LOAD 0x00000000307f0e30 0x0000000000000000 0x00000000fffe0000
> + LOAD 0x00000000307f01ac 0x0000000000000000 0x00000000fffe0000
> 0x0000000000020000 0x0000000000020000 0
>
> There is no dynamic section in this file.
> @@ -47,13 +47,6 @@
>
> No version information found in this file.
>
> -Notes at offset 0x00000190 with length 0x00000ca0:
> +Notes at offset 0x00000190 with length 0x0000001c:
> Owner Data size Description
> - CORE 0x00000150 NT_PRSTATUS (prstatus structure)
> - CORE 0x00000150 NT_PRSTATUS (prstatus structure)
> - CORE 0x00000150 NT_PRSTATUS (prstatus structure)
> - CORE 0x00000150 NT_PRSTATUS (prstatus structure)
> - QEMU 0x000001b0 Unknown note type: (0x00000000)
> - QEMU 0x000001b0 Unknown note type: (0x00000000)
> - QEMU 0x000001b0 Unknown note type: (0x00000000)
> - QEMU 0x000001b0 Unknown note type: (0x00000000)
> + NONE 0x00000005 Unknown note type: (0x00000000)
- The vmcore was checked with "crash" too, again comparing the results
against a vmcore written by qmp_dump_guest_memory():
> --- guest.vmcore.log2 2013-09-12 17:52:27.074289201 +0200
> +++ example.dump.log2 2013-09-12 17:52:15.904289203 +0200
> @@ -22,11 +22,11 @@
> This GDB was configured as "x86_64-unknown-linux-gnu"...
>
> KERNEL: /usr/lib/debug/lib/modules/2.6.32-358.el6.x86_64/vmlinux
> - DUMPFILE: /home/lacos/tmp/guest.vmcore
> + DUMPFILE: /home/lacos/tmp/example.dump
> CPUS: 4
> - DATE: Thu Sep 12 17:16:11 2013
> - UPTIME: 00:01:09
> -LOAD AVERAGE: 0.07, 0.03, 0.00
> + DATE: Thu Sep 12 17:17:41 2013
> + UPTIME: 00:00:38
> +LOAD AVERAGE: 0.18, 0.05, 0.01
> TASKS: 130
> NODENAME: localhost.localdomain
> RELEASE: 2.6.32-358.el6.x86_64
> @@ -38,12 +38,12 @@
> COMMAND: "swapper"
> TASK: ffffffff81a8d020 (1 of 4) [THREAD_INFO: ffffffff81a00000]
> CPU: 0
> - STATE: TASK_RUNNING (PANIC)
> + STATE: TASK_RUNNING (ACTIVE)
> + WARNING: panic task not found
>
> crash> bt
> PID: 0 TASK: ffffffff81a8d020 CPU: 0 COMMAND: "swapper"
> - #0 [ffffffff81a01ed0] default_idle at ffffffff8101495d
> - #1 [ffffffff81a01ef0] cpu_idle at ffffffff81009fc6
> + #0 [ffffffff81a01ef0] cpu_idle at ffffffff81009fc6
> crash> task ffffffff81a8d020
> PID: 0 TASK: ffffffff81a8d020 CPU: 0 COMMAND: "swapper"
> struct task_struct {
> @@ -75,7 +75,7 @@
> prev = 0xffffffff81a8d080
> },
> on_rq = 0,
> - exec_start = 8618466836,
> + exec_start = 7469214014,
> sum_exec_runtime = 0,
> vruntime = 0,
> prev_sum_exec_runtime = 0,
> @@ -149,7 +149,7 @@
> },
> tasks = {
> next = 0xffff88002d621948,
> - prev = 0xffff880029618f28
> + prev = 0xffff880023b74488
> },
> pushable_tasks = {
> prio = 140,
> @@ -165,7 +165,7 @@
> }
> },
> mm = 0x0,
> - active_mm = 0xffff88002929b780,
> + active_mm = 0xffff8800297eb980,
> exit_state = 0,
> exit_code = 0,
> exit_signal = 0,
> @@ -177,7 +177,7 @@
> sched_reset_on_fork = 0,
> pid = 0,
> tgid = 0,
> - stack_canary = 2483693585637059287,
> + stack_canary = 7266362296181431986,
> real_parent = 0xffffffff81a8d020,
> parent = 0xffffffff81a8d020,
> children = {
> @@ -224,14 +224,14 @@
> set_child_tid = 0x0,
> clear_child_tid = 0x0,
> utime = 0,
> - stime = 3,
> + stime = 2,
> utimescaled = 0,
> - stimescaled = 3,
> + stimescaled = 2,
> gtime = 0,
> prev_utime = 0,
> prev_stime = 0,
> nvcsw = 0,
> - nivcsw = 1000,
> + nivcsw = 1764,
> start_time = {
> tv_sec = 0,
> tv_nsec = 0
- <name_dropping>I asked for Dave Anderson's help with verifying the
extracted vmcore, and his comments make me think I should post
this.</name_dropping>
Signed-off-by: Laszlo Ersek <lersek@redhat.com>
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
3e16d14fd9