OpenE2K
/
qemu-e2k
13
4
Fork 0
Code Issues 4 Pull Requests Projects 2 Releases Activity
qemu-e2k/tests/qemu-iotests/179.out

202 lines
15 KiB
Plaintext
Raw Normal View History

iotests: Add test 179 to cover write zeroes with unmap No tests were covering write zeroes with unmap. Additionally, I needed to prove that my previous patches for correct status reporting and write zeroes optimizations actually had an impact. The test works for cluster_size between 8k and 2M (for smaller sizes, it fails because our allocation patterns are not contiguous with small clusters - in part, the largest consecutive allocation we tend to get is often bounded by the size covered by one L2 table). Note that testing for zero clusters is tricky: 'qemu-io map' reports whether data comes from the current layer of the image (useful for sniffing out which regions of the file have QCOW_OFLAG_ZERO) - but doesn't show which clusters have mappings; while 'qemu-img map' sees "zero":true for both unallocated and zero clusters for any qcow2 with no backing layer (so less useful at detecting true zero clusters), but reliably shows mappings. So we have to rely on both queries side-by-side at each point of the test. Signed-off-by: Eric Blake <eblake@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-id: 20170507000552.20847-10-eblake@redhat.com Signed-off-by: Max Reitz <mreitz@redhat.com>
2017-05-07 02:05:49 +02:00
QA output created by 179
=== Testing write zeroes with unmap ===
Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=67108864
iotests: Specify explicit backing format where sensible There are many existing qcow2 images that specify a backing file but no format. This has been the source of CVEs in the past, but has become more prominent of a problem now that libvirt has switched to -blockdev. With older -drive, at least the probing was always done by qemu (so the only risk of a changed format between successive boots of a guest was if qemu was upgraded and probed differently). But with newer -blockdev, libvirt must specify a format; if libvirt guesses raw where the image was formatted, this results in data corruption visible to the guest; conversely, if libvirt guesses qcow2 where qemu was using raw, this can result in potential security holes, so modern libvirt instead refuses to use images without explicit backing format. The change in libvirt to reject images without explicit backing format has pointed out that a number of tools have been far too reliant on probing in the past. It's time to set a better example in our own iotests of properly setting this parameter. iotest calls to create, rebase, and convert are all impacted to some degree. It's a bit annoying that we are inconsistent on command line - while all of those accept -o backing_file=...,backing_fmt=..., the shortcuts are different: create and rebase have -b and -F, while convert has -B but no -F. (amend has no shortcuts, but the previous patch just deprecated the use of amend to change backing chains). Signed-off-by: Eric Blake <eblake@redhat.com> Message-Id: <20200706203954.341758-9-eblake@redhat.com> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-06 22:39:52 +02:00
Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 backing_file=TEST_DIR/t.IMGFMT.base backing_fmt=IMGFMT
iotests: Add test 179 to cover write zeroes with unmap No tests were covering write zeroes with unmap. Additionally, I needed to prove that my previous patches for correct status reporting and write zeroes optimizations actually had an impact. The test works for cluster_size between 8k and 2M (for smaller sizes, it fails because our allocation patterns are not contiguous with small clusters - in part, the largest consecutive allocation we tend to get is often bounded by the size covered by one L2 table). Note that testing for zero clusters is tricky: 'qemu-io map' reports whether data comes from the current layer of the image (useful for sniffing out which regions of the file have QCOW_OFLAG_ZERO) - but doesn't show which clusters have mappings; while 'qemu-img map' sees "zero":true for both unallocated and zero clusters for any qcow2 with no backing layer (so less useful at detecting true zero clusters), but reliably shows mappings. So we have to rely on both queries side-by-side at each point of the test. Signed-off-by: Eric Blake <eblake@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-id: 20170507000552.20847-10-eblake@redhat.com Signed-off-by: Max Reitz <mreitz@redhat.com>
2017-05-07 02:05:49 +02:00
wrote 2097152/2097152 bytes at offset 2097152
2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
wrote 2097152/2097152 bytes at offset 6291456
2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
2 MiB (0x200000) bytes not allocated at offset 0 bytes (0x0)
2 MiB (0x200000) bytes allocated at offset 2 MiB (0x200000)
2 MiB (0x200000) bytes not allocated at offset 4 MiB (0x400000)
2 MiB (0x200000) bytes allocated at offset 6 MiB (0x600000)
56 MiB (0x3800000) bytes not allocated at offset 8 MiB (0x800000)
qemu-img: Make unallocated part of backing chain obvious in map The recently-added NBD context qemu:allocation-depth is able to distinguish between locally-present data (even when that data is sparse) [shown as depth 1 over NBD], and data that could not be found anywhere in the backing chain [shown as depth 0]; and the libnbd project was recently patched to give the human-readable name "absent" to an allocation-depth of 0. But qemu-img map --output=json predates that addition, and has the unfortunate behavior that all portions of the backing chain that resolve without finding a hit in any backing layer report the same depth as the final backing layer. This makes it harder to reconstruct a qcow2 backing chain using just 'qemu-img map' output, especially when using "backing":null to artificially limit a backing chain, because it is impossible to distinguish between a QCOW2_CLUSTER_UNALLOCATED (which defers to a [missing] backing file) and a QCOW2_CLUSTER_ZERO_PLAIN cluster (which would override any backing file), since both types of clusters otherwise show as "data":false,"zero":true" (but note that we can distinguish a QCOW2_CLUSTER_ZERO_ALLOCATED, which would also have an "offset": listing). The task of reconstructing a qcow2 chain was made harder in commit 0da9856851 (nbd: server: Report holes for raw images), because prior to that point, it was possible to abuse NBD's block status command to see which portions of a qcow2 file resulted in BDRV_BLOCK_ALLOCATED (showing up as NBD_STATE_ZERO in isolation) vs. missing from the chain (showing up as NBD_STATE_ZERO|NBD_STATE_HOLE); but now qemu reports more accurate sparseness information over NBD. An obvious solution is to make 'qemu-img map --output=json' add an additional "present":false designation to any cluster lacking an allocation anywhere in the chain, without any change to the "depth" parameter to avoid breaking existing clients. The iotests have several examples where this distinction demonstrates the additional accuracy. Signed-off-by: Eric Blake <eblake@redhat.com> Message-Id: <20210701190655.2131223-3-eblake@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> [eblake: fix more iotest fallout] Signed-off-by: Eric Blake <eblake@redhat.com>
2021-07-01 21:06:55 +02:00
[{ "start": 0, "length": 2097152, "depth": 0, "present": false, "zero": true, "data": false},
{ "start": 2097152, "length": 2097152, "depth": 0, "present": true, "zero": true, "data": false},
{ "start": 4194304, "length": 2097152, "depth": 0, "present": false, "zero": true, "data": false},
{ "start": 6291456, "length": 2097152, "depth": 0, "present": true, "zero": true, "data": false},
{ "start": 8388608, "length": 58720256, "depth": 0, "present": false, "zero": true, "data": false}]
iotests: Add test 179 to cover write zeroes with unmap No tests were covering write zeroes with unmap. Additionally, I needed to prove that my previous patches for correct status reporting and write zeroes optimizations actually had an impact. The test works for cluster_size between 8k and 2M (for smaller sizes, it fails because our allocation patterns are not contiguous with small clusters - in part, the largest consecutive allocation we tend to get is often bounded by the size covered by one L2 table). Note that testing for zero clusters is tricky: 'qemu-io map' reports whether data comes from the current layer of the image (useful for sniffing out which regions of the file have QCOW_OFLAG_ZERO) - but doesn't show which clusters have mappings; while 'qemu-img map' sees "zero":true for both unallocated and zero clusters for any qcow2 with no backing layer (so less useful at detecting true zero clusters), but reliably shows mappings. So we have to rely on both queries side-by-side at each point of the test. Signed-off-by: Eric Blake <eblake@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-id: 20170507000552.20847-10-eblake@redhat.com Signed-off-by: Max Reitz <mreitz@redhat.com>
2017-05-07 02:05:49 +02:00
wrote 2097150/2097150 bytes at offset 10485761
2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
wrote 2097150/2097150 bytes at offset 14680065
2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
2 MiB (0x200000) bytes not allocated at offset 0 bytes (0x0)
2 MiB (0x200000) bytes allocated at offset 2 MiB (0x200000)
2 MiB (0x200000) bytes not allocated at offset 4 MiB (0x400000)
2 MiB (0x200000) bytes allocated at offset 6 MiB (0x600000)
2 MiB (0x200000) bytes not allocated at offset 8 MiB (0x800000)
2 MiB (0x200000) bytes allocated at offset 10 MiB (0xa00000)
2 MiB (0x200000) bytes not allocated at offset 12 MiB (0xc00000)
2 MiB (0x200000) bytes allocated at offset 14 MiB (0xe00000)
48 MiB (0x3000000) bytes not allocated at offset 16 MiB (0x1000000)
qemu-img: Make unallocated part of backing chain obvious in map The recently-added NBD context qemu:allocation-depth is able to distinguish between locally-present data (even when that data is sparse) [shown as depth 1 over NBD], and data that could not be found anywhere in the backing chain [shown as depth 0]; and the libnbd project was recently patched to give the human-readable name "absent" to an allocation-depth of 0. But qemu-img map --output=json predates that addition, and has the unfortunate behavior that all portions of the backing chain that resolve without finding a hit in any backing layer report the same depth as the final backing layer. This makes it harder to reconstruct a qcow2 backing chain using just 'qemu-img map' output, especially when using "backing":null to artificially limit a backing chain, because it is impossible to distinguish between a QCOW2_CLUSTER_UNALLOCATED (which defers to a [missing] backing file) and a QCOW2_CLUSTER_ZERO_PLAIN cluster (which would override any backing file), since both types of clusters otherwise show as "data":false,"zero":true" (but note that we can distinguish a QCOW2_CLUSTER_ZERO_ALLOCATED, which would also have an "offset": listing). The task of reconstructing a qcow2 chain was made harder in commit 0da9856851 (nbd: server: Report holes for raw images), because prior to that point, it was possible to abuse NBD's block status command to see which portions of a qcow2 file resulted in BDRV_BLOCK_ALLOCATED (showing up as NBD_STATE_ZERO in isolation) vs. missing from the chain (showing up as NBD_STATE_ZERO|NBD_STATE_HOLE); but now qemu reports more accurate sparseness information over NBD. An obvious solution is to make 'qemu-img map --output=json' add an additional "present":false designation to any cluster lacking an allocation anywhere in the chain, without any change to the "depth" parameter to avoid breaking existing clients. The iotests have several examples where this distinction demonstrates the additional accuracy. Signed-off-by: Eric Blake <eblake@redhat.com> Message-Id: <20210701190655.2131223-3-eblake@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> [eblake: fix more iotest fallout] Signed-off-by: Eric Blake <eblake@redhat.com>
2021-07-01 21:06:55 +02:00
[{ "start": 0, "length": 2097152, "depth": 0, "present": false, "zero": true, "data": false},
{ "start": 2097152, "length": 2097152, "depth": 0, "present": true, "zero": true, "data": false},
{ "start": 4194304, "length": 2097152, "depth": 0, "present": false, "zero": true, "data": false},
{ "start": 6291456, "length": 2097152, "depth": 0, "present": true, "zero": true, "data": false},
{ "start": 8388608, "length": 2097152, "depth": 0, "present": false, "zero": true, "data": false},
{ "start": 10485760, "length": 2097152, "depth": 0, "present": true, "zero": true, "data": false},
{ "start": 12582912, "length": 2097152, "depth": 0, "present": false, "zero": true, "data": false},
{ "start": 14680064, "length": 2097152, "depth": 0, "present": true, "zero": true, "data": false},
{ "start": 16777216, "length": 50331648, "depth": 0, "present": false, "zero": true, "data": false}]
iotests: Add test 179 to cover write zeroes with unmap No tests were covering write zeroes with unmap. Additionally, I needed to prove that my previous patches for correct status reporting and write zeroes optimizations actually had an impact. The test works for cluster_size between 8k and 2M (for smaller sizes, it fails because our allocation patterns are not contiguous with small clusters - in part, the largest consecutive allocation we tend to get is often bounded by the size covered by one L2 table). Note that testing for zero clusters is tricky: 'qemu-io map' reports whether data comes from the current layer of the image (useful for sniffing out which regions of the file have QCOW_OFLAG_ZERO) - but doesn't show which clusters have mappings; while 'qemu-img map' sees "zero":true for both unallocated and zero clusters for any qcow2 with no backing layer (so less useful at detecting true zero clusters), but reliably shows mappings. So we have to rely on both queries side-by-side at each point of the test. Signed-off-by: Eric Blake <eblake@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-id: 20170507000552.20847-10-eblake@redhat.com Signed-off-by: Max Reitz <mreitz@redhat.com>
2017-05-07 02:05:49 +02:00
wrote 14680064/14680064 bytes at offset 18874368
14 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
wrote 2097152/2097152 bytes at offset 20971520
2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
wrote 6291456/6291456 bytes at offset 25165824
6 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
2 MiB (0x200000) bytes not allocated at offset 0 bytes (0x0)
2 MiB (0x200000) bytes allocated at offset 2 MiB (0x200000)
2 MiB (0x200000) bytes not allocated at offset 4 MiB (0x400000)
2 MiB (0x200000) bytes allocated at offset 6 MiB (0x600000)
2 MiB (0x200000) bytes not allocated at offset 8 MiB (0x800000)
2 MiB (0x200000) bytes allocated at offset 10 MiB (0xa00000)
2 MiB (0x200000) bytes not allocated at offset 12 MiB (0xc00000)
2 MiB (0x200000) bytes allocated at offset 14 MiB (0xe00000)
2 MiB (0x200000) bytes not allocated at offset 16 MiB (0x1000000)
14 MiB (0xe00000) bytes allocated at offset 18 MiB (0x1200000)
32 MiB (0x2000000) bytes not allocated at offset 32 MiB (0x2000000)
qemu-img: Make unallocated part of backing chain obvious in map The recently-added NBD context qemu:allocation-depth is able to distinguish between locally-present data (even when that data is sparse) [shown as depth 1 over NBD], and data that could not be found anywhere in the backing chain [shown as depth 0]; and the libnbd project was recently patched to give the human-readable name "absent" to an allocation-depth of 0. But qemu-img map --output=json predates that addition, and has the unfortunate behavior that all portions of the backing chain that resolve without finding a hit in any backing layer report the same depth as the final backing layer. This makes it harder to reconstruct a qcow2 backing chain using just 'qemu-img map' output, especially when using "backing":null to artificially limit a backing chain, because it is impossible to distinguish between a QCOW2_CLUSTER_UNALLOCATED (which defers to a [missing] backing file) and a QCOW2_CLUSTER_ZERO_PLAIN cluster (which would override any backing file), since both types of clusters otherwise show as "data":false,"zero":true" (but note that we can distinguish a QCOW2_CLUSTER_ZERO_ALLOCATED, which would also have an "offset": listing). The task of reconstructing a qcow2 chain was made harder in commit 0da9856851 (nbd: server: Report holes for raw images), because prior to that point, it was possible to abuse NBD's block status command to see which portions of a qcow2 file resulted in BDRV_BLOCK_ALLOCATED (showing up as NBD_STATE_ZERO in isolation) vs. missing from the chain (showing up as NBD_STATE_ZERO|NBD_STATE_HOLE); but now qemu reports more accurate sparseness information over NBD. An obvious solution is to make 'qemu-img map --output=json' add an additional "present":false designation to any cluster lacking an allocation anywhere in the chain, without any change to the "depth" parameter to avoid breaking existing clients. The iotests have several examples where this distinction demonstrates the additional accuracy. Signed-off-by: Eric Blake <eblake@redhat.com> Message-Id: <20210701190655.2131223-3-eblake@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> [eblake: fix more iotest fallout] Signed-off-by: Eric Blake <eblake@redhat.com>
2021-07-01 21:06:55 +02:00
[{ "start": 0, "length": 2097152, "depth": 0, "present": false, "zero": true, "data": false},
{ "start": 2097152, "length": 2097152, "depth": 0, "present": true, "zero": true, "data": false},
{ "start": 4194304, "length": 2097152, "depth": 0, "present": false, "zero": true, "data": false},
{ "start": 6291456, "length": 2097152, "depth": 0, "present": true, "zero": true, "data": false},
{ "start": 8388608, "length": 2097152, "depth": 0, "present": false, "zero": true, "data": false},
{ "start": 10485760, "length": 2097152, "depth": 0, "present": true, "zero": true, "data": false},
{ "start": 12582912, "length": 2097152, "depth": 0, "present": false, "zero": true, "data": false},
{ "start": 14680064, "length": 2097152, "depth": 0, "present": true, "zero": true, "data": false},
{ "start": 16777216, "length": 2097152, "depth": 0, "present": false, "zero": true, "data": false},
{ "start": 18874368, "length": 2097152, "depth": 0, "present": true, "zero": false, "data": true, "offset": OFFSET},
{ "start": 20971520, "length": 2097152, "depth": 0, "present": true, "zero": true, "data": false},
{ "start": 23068672, "length": 2097152, "depth": 0, "present": true, "zero": false, "data": true, "offset": OFFSET},
{ "start": 25165824, "length": 6291456, "depth": 0, "present": true, "zero": true, "data": false, "offset": OFFSET},
{ "start": 31457280, "length": 2097152, "depth": 0, "present": true, "zero": false, "data": true, "offset": OFFSET},
{ "start": 33554432, "length": 33554432, "depth": 0, "present": false, "zero": true, "data": false}]
iotests: Add test 179 to cover write zeroes with unmap No tests were covering write zeroes with unmap. Additionally, I needed to prove that my previous patches for correct status reporting and write zeroes optimizations actually had an impact. The test works for cluster_size between 8k and 2M (for smaller sizes, it fails because our allocation patterns are not contiguous with small clusters - in part, the largest consecutive allocation we tend to get is often bounded by the size covered by one L2 table). Note that testing for zero clusters is tricky: 'qemu-io map' reports whether data comes from the current layer of the image (useful for sniffing out which regions of the file have QCOW_OFLAG_ZERO) - but doesn't show which clusters have mappings; while 'qemu-img map' sees "zero":true for both unallocated and zero clusters for any qcow2 with no backing layer (so less useful at detecting true zero clusters), but reliably shows mappings. So we have to rely on both queries side-by-side at each point of the test. Signed-off-by: Eric Blake <eblake@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-id: 20170507000552.20847-10-eblake@redhat.com Signed-off-by: Max Reitz <mreitz@redhat.com>
2017-05-07 02:05:49 +02:00
wrote 2097152/2097152 bytes at offset 27262976
2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
wrote 2097152/2097152 bytes at offset 29360128
2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
2 MiB (0x200000) bytes not allocated at offset 0 bytes (0x0)
2 MiB (0x200000) bytes allocated at offset 2 MiB (0x200000)
2 MiB (0x200000) bytes not allocated at offset 4 MiB (0x400000)
2 MiB (0x200000) bytes allocated at offset 6 MiB (0x600000)
2 MiB (0x200000) bytes not allocated at offset 8 MiB (0x800000)
2 MiB (0x200000) bytes allocated at offset 10 MiB (0xa00000)
2 MiB (0x200000) bytes not allocated at offset 12 MiB (0xc00000)
2 MiB (0x200000) bytes allocated at offset 14 MiB (0xe00000)
2 MiB (0x200000) bytes not allocated at offset 16 MiB (0x1000000)
14 MiB (0xe00000) bytes allocated at offset 18 MiB (0x1200000)
32 MiB (0x2000000) bytes not allocated at offset 32 MiB (0x2000000)
qemu-img: Make unallocated part of backing chain obvious in map The recently-added NBD context qemu:allocation-depth is able to distinguish between locally-present data (even when that data is sparse) [shown as depth 1 over NBD], and data that could not be found anywhere in the backing chain [shown as depth 0]; and the libnbd project was recently patched to give the human-readable name "absent" to an allocation-depth of 0. But qemu-img map --output=json predates that addition, and has the unfortunate behavior that all portions of the backing chain that resolve without finding a hit in any backing layer report the same depth as the final backing layer. This makes it harder to reconstruct a qcow2 backing chain using just 'qemu-img map' output, especially when using "backing":null to artificially limit a backing chain, because it is impossible to distinguish between a QCOW2_CLUSTER_UNALLOCATED (which defers to a [missing] backing file) and a QCOW2_CLUSTER_ZERO_PLAIN cluster (which would override any backing file), since both types of clusters otherwise show as "data":false,"zero":true" (but note that we can distinguish a QCOW2_CLUSTER_ZERO_ALLOCATED, which would also have an "offset": listing). The task of reconstructing a qcow2 chain was made harder in commit 0da9856851 (nbd: server: Report holes for raw images), because prior to that point, it was possible to abuse NBD's block status command to see which portions of a qcow2 file resulted in BDRV_BLOCK_ALLOCATED (showing up as NBD_STATE_ZERO in isolation) vs. missing from the chain (showing up as NBD_STATE_ZERO|NBD_STATE_HOLE); but now qemu reports more accurate sparseness information over NBD. An obvious solution is to make 'qemu-img map --output=json' add an additional "present":false designation to any cluster lacking an allocation anywhere in the chain, without any change to the "depth" parameter to avoid breaking existing clients. The iotests have several examples where this distinction demonstrates the additional accuracy. Signed-off-by: Eric Blake <eblake@redhat.com> Message-Id: <20210701190655.2131223-3-eblake@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> [eblake: fix more iotest fallout] Signed-off-by: Eric Blake <eblake@redhat.com>
2021-07-01 21:06:55 +02:00
[{ "start": 0, "length": 2097152, "depth": 0, "present": false, "zero": true, "data": false},
{ "start": 2097152, "length": 2097152, "depth": 0, "present": true, "zero": true, "data": false},
{ "start": 4194304, "length": 2097152, "depth": 0, "present": false, "zero": true, "data": false},
{ "start": 6291456, "length": 2097152, "depth": 0, "present": true, "zero": true, "data": false},
{ "start": 8388608, "length": 2097152, "depth": 0, "present": false, "zero": true, "data": false},
{ "start": 10485760, "length": 2097152, "depth": 0, "present": true, "zero": true, "data": false},
{ "start": 12582912, "length": 2097152, "depth": 0, "present": false, "zero": true, "data": false},
{ "start": 14680064, "length": 2097152, "depth": 0, "present": true, "zero": true, "data": false},
{ "start": 16777216, "length": 2097152, "depth": 0, "present": false, "zero": true, "data": false},
{ "start": 18874368, "length": 2097152, "depth": 0, "present": true, "zero": false, "data": true, "offset": OFFSET},
{ "start": 20971520, "length": 2097152, "depth": 0, "present": true, "zero": true, "data": false},
{ "start": 23068672, "length": 2097152, "depth": 0, "present": true, "zero": false, "data": true, "offset": OFFSET},
{ "start": 25165824, "length": 2097152, "depth": 0, "present": true, "zero": true, "data": false, "offset": OFFSET},
{ "start": 27262976, "length": 2097152, "depth": 0, "present": true, "zero": true, "data": false},
{ "start": 29360128, "length": 2097152, "depth": 0, "present": true, "zero": true, "data": false, "offset": OFFSET},
{ "start": 31457280, "length": 2097152, "depth": 0, "present": true, "zero": false, "data": true, "offset": OFFSET},
{ "start": 33554432, "length": 33554432, "depth": 0, "present": false, "zero": true, "data": false}]
iotests: Add test 179 to cover write zeroes with unmap No tests were covering write zeroes with unmap. Additionally, I needed to prove that my previous patches for correct status reporting and write zeroes optimizations actually had an impact. The test works for cluster_size between 8k and 2M (for smaller sizes, it fails because our allocation patterns are not contiguous with small clusters - in part, the largest consecutive allocation we tend to get is often bounded by the size covered by one L2 table). Note that testing for zero clusters is tricky: 'qemu-io map' reports whether data comes from the current layer of the image (useful for sniffing out which regions of the file have QCOW_OFLAG_ZERO) - but doesn't show which clusters have mappings; while 'qemu-img map' sees "zero":true for both unallocated and zero clusters for any qcow2 with no backing layer (so less useful at detecting true zero clusters), but reliably shows mappings. So we have to rely on both queries side-by-side at each point of the test. Signed-off-by: Eric Blake <eblake@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-id: 20170507000552.20847-10-eblake@redhat.com Signed-off-by: Max Reitz <mreitz@redhat.com>
2017-05-07 02:05:49 +02:00
wrote 8388608/8388608 bytes at offset 33554432
8 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
wrote 2097152/2097152 bytes at offset 35651584
2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
wrote 2097152/2097152 bytes at offset 37748736
2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
2 MiB (0x200000) bytes not allocated at offset 0 bytes (0x0)
2 MiB (0x200000) bytes allocated at offset 2 MiB (0x200000)
2 MiB (0x200000) bytes not allocated at offset 4 MiB (0x400000)
2 MiB (0x200000) bytes allocated at offset 6 MiB (0x600000)
2 MiB (0x200000) bytes not allocated at offset 8 MiB (0x800000)
2 MiB (0x200000) bytes allocated at offset 10 MiB (0xa00000)
2 MiB (0x200000) bytes not allocated at offset 12 MiB (0xc00000)
2 MiB (0x200000) bytes allocated at offset 14 MiB (0xe00000)
2 MiB (0x200000) bytes not allocated at offset 16 MiB (0x1000000)
22 MiB (0x1600000) bytes allocated at offset 18 MiB (0x1200000)
24 MiB (0x1800000) bytes not allocated at offset 40 MiB (0x2800000)
qemu-img: Make unallocated part of backing chain obvious in map The recently-added NBD context qemu:allocation-depth is able to distinguish between locally-present data (even when that data is sparse) [shown as depth 1 over NBD], and data that could not be found anywhere in the backing chain [shown as depth 0]; and the libnbd project was recently patched to give the human-readable name "absent" to an allocation-depth of 0. But qemu-img map --output=json predates that addition, and has the unfortunate behavior that all portions of the backing chain that resolve without finding a hit in any backing layer report the same depth as the final backing layer. This makes it harder to reconstruct a qcow2 backing chain using just 'qemu-img map' output, especially when using "backing":null to artificially limit a backing chain, because it is impossible to distinguish between a QCOW2_CLUSTER_UNALLOCATED (which defers to a [missing] backing file) and a QCOW2_CLUSTER_ZERO_PLAIN cluster (which would override any backing file), since both types of clusters otherwise show as "data":false,"zero":true" (but note that we can distinguish a QCOW2_CLUSTER_ZERO_ALLOCATED, which would also have an "offset": listing). The task of reconstructing a qcow2 chain was made harder in commit 0da9856851 (nbd: server: Report holes for raw images), because prior to that point, it was possible to abuse NBD's block status command to see which portions of a qcow2 file resulted in BDRV_BLOCK_ALLOCATED (showing up as NBD_STATE_ZERO in isolation) vs. missing from the chain (showing up as NBD_STATE_ZERO|NBD_STATE_HOLE); but now qemu reports more accurate sparseness information over NBD. An obvious solution is to make 'qemu-img map --output=json' add an additional "present":false designation to any cluster lacking an allocation anywhere in the chain, without any change to the "depth" parameter to avoid breaking existing clients. The iotests have several examples where this distinction demonstrates the additional accuracy. Signed-off-by: Eric Blake <eblake@redhat.com> Message-Id: <20210701190655.2131223-3-eblake@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> [eblake: fix more iotest fallout] Signed-off-by: Eric Blake <eblake@redhat.com>
2021-07-01 21:06:55 +02:00
[{ "start": 0, "length": 2097152, "depth": 0, "present": false, "zero": true, "data": false},
{ "start": 2097152, "length": 2097152, "depth": 0, "present": true, "zero": true, "data": false},
{ "start": 4194304, "length": 2097152, "depth": 0, "present": false, "zero": true, "data": false},
{ "start": 6291456, "length": 2097152, "depth": 0, "present": true, "zero": true, "data": false},
{ "start": 8388608, "length": 2097152, "depth": 0, "present": false, "zero": true, "data": false},
{ "start": 10485760, "length": 2097152, "depth": 0, "present": true, "zero": true, "data": false},
{ "start": 12582912, "length": 2097152, "depth": 0, "present": false, "zero": true, "data": false},
{ "start": 14680064, "length": 2097152, "depth": 0, "present": true, "zero": true, "data": false},
{ "start": 16777216, "length": 2097152, "depth": 0, "present": false, "zero": true, "data": false},
{ "start": 18874368, "length": 2097152, "depth": 0, "present": true, "zero": false, "data": true, "offset": OFFSET},
{ "start": 20971520, "length": 2097152, "depth": 0, "present": true, "zero": true, "data": false},
{ "start": 23068672, "length": 2097152, "depth": 0, "present": true, "zero": false, "data": true, "offset": OFFSET},
{ "start": 25165824, "length": 2097152, "depth": 0, "present": true, "zero": true, "data": false, "offset": OFFSET},
{ "start": 27262976, "length": 2097152, "depth": 0, "present": true, "zero": true, "data": false},
{ "start": 29360128, "length": 2097152, "depth": 0, "present": true, "zero": true, "data": false, "offset": OFFSET},
{ "start": 31457280, "length": 2097152, "depth": 0, "present": true, "zero": false, "data": true, "offset": OFFSET},
{ "start": 33554432, "length": 8388608, "depth": 0, "present": true, "zero": true, "data": false},
{ "start": 41943040, "length": 25165824, "depth": 0, "present": false, "zero": true, "data": false}]
iotests: Add test 179 to cover write zeroes with unmap No tests were covering write zeroes with unmap. Additionally, I needed to prove that my previous patches for correct status reporting and write zeroes optimizations actually had an impact. The test works for cluster_size between 8k and 2M (for smaller sizes, it fails because our allocation patterns are not contiguous with small clusters - in part, the largest consecutive allocation we tend to get is often bounded by the size covered by one L2 table). Note that testing for zero clusters is tricky: 'qemu-io map' reports whether data comes from the current layer of the image (useful for sniffing out which regions of the file have QCOW_OFLAG_ZERO) - but doesn't show which clusters have mappings; while 'qemu-img map' sees "zero":true for both unallocated and zero clusters for any qcow2 with no backing layer (so less useful at detecting true zero clusters), but reliably shows mappings. So we have to rely on both queries side-by-side at each point of the test. Signed-off-by: Eric Blake <eblake@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-id: 20170507000552.20847-10-eblake@redhat.com Signed-off-by: Max Reitz <mreitz@redhat.com>
2017-05-07 02:05:49 +02:00
wrote 8388608/8388608 bytes at offset 41943040
8 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
wrote 8388608/8388608 bytes at offset 50331648
8 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
wrote 2097152/2097152 bytes at offset 44040192
2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
wrote 2097152/2097152 bytes at offset 46137344
2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
wrote 2097152/2097152 bytes at offset 52428800
2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
wrote 2097152/2097152 bytes at offset 54525952
2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
wrote 2097152/2097152 bytes at offset 60817408
2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
wrote 2097152/2097152 bytes at offset 62914560
2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
42 MiB (0x2a00000) bytes not allocated at offset 0 bytes (0x0)
4 MiB (0x400000) bytes allocated at offset 42 MiB (0x2a00000)
4 MiB (0x400000) bytes not allocated at offset 46 MiB (0x2e00000)
4 MiB (0x400000) bytes allocated at offset 50 MiB (0x3200000)
4 MiB (0x400000) bytes not allocated at offset 54 MiB (0x3600000)
4 MiB (0x400000) bytes allocated at offset 58 MiB (0x3a00000)
2 MiB (0x200000) bytes not allocated at offset 62 MiB (0x3e00000)
qemu-img: Make unallocated part of backing chain obvious in map The recently-added NBD context qemu:allocation-depth is able to distinguish between locally-present data (even when that data is sparse) [shown as depth 1 over NBD], and data that could not be found anywhere in the backing chain [shown as depth 0]; and the libnbd project was recently patched to give the human-readable name "absent" to an allocation-depth of 0. But qemu-img map --output=json predates that addition, and has the unfortunate behavior that all portions of the backing chain that resolve without finding a hit in any backing layer report the same depth as the final backing layer. This makes it harder to reconstruct a qcow2 backing chain using just 'qemu-img map' output, especially when using "backing":null to artificially limit a backing chain, because it is impossible to distinguish between a QCOW2_CLUSTER_UNALLOCATED (which defers to a [missing] backing file) and a QCOW2_CLUSTER_ZERO_PLAIN cluster (which would override any backing file), since both types of clusters otherwise show as "data":false,"zero":true" (but note that we can distinguish a QCOW2_CLUSTER_ZERO_ALLOCATED, which would also have an "offset": listing). The task of reconstructing a qcow2 chain was made harder in commit 0da9856851 (nbd: server: Report holes for raw images), because prior to that point, it was possible to abuse NBD's block status command to see which portions of a qcow2 file resulted in BDRV_BLOCK_ALLOCATED (showing up as NBD_STATE_ZERO in isolation) vs. missing from the chain (showing up as NBD_STATE_ZERO|NBD_STATE_HOLE); but now qemu reports more accurate sparseness information over NBD. An obvious solution is to make 'qemu-img map --output=json' add an additional "present":false designation to any cluster lacking an allocation anywhere in the chain, without any change to the "depth" parameter to avoid breaking existing clients. The iotests have several examples where this distinction demonstrates the additional accuracy. Signed-off-by: Eric Blake <eblake@redhat.com> Message-Id: <20210701190655.2131223-3-eblake@redhat.com> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> [eblake: fix more iotest fallout] Signed-off-by: Eric Blake <eblake@redhat.com>
2021-07-01 21:06:55 +02:00
[{ "start": 0, "length": 2097152, "depth": 1, "present": false, "zero": true, "data": false},
{ "start": 2097152, "length": 2097152, "depth": 1, "present": true, "zero": true, "data": false},
{ "start": 4194304, "length": 2097152, "depth": 1, "present": false, "zero": true, "data": false},
{ "start": 6291456, "length": 2097152, "depth": 1, "present": true, "zero": true, "data": false},
{ "start": 8388608, "length": 2097152, "depth": 1, "present": false, "zero": true, "data": false},
{ "start": 10485760, "length": 2097152, "depth": 1, "present": true, "zero": true, "data": false},
{ "start": 12582912, "length": 2097152, "depth": 1, "present": false, "zero": true, "data": false},
{ "start": 14680064, "length": 2097152, "depth": 1, "present": true, "zero": true, "data": false},
{ "start": 16777216, "length": 2097152, "depth": 1, "present": false, "zero": true, "data": false},
{ "start": 18874368, "length": 2097152, "depth": 1, "present": true, "zero": false, "data": true, "offset": OFFSET},
{ "start": 20971520, "length": 2097152, "depth": 1, "present": true, "zero": true, "data": false},
{ "start": 23068672, "length": 2097152, "depth": 1, "present": true, "zero": false, "data": true, "offset": OFFSET},
{ "start": 25165824, "length": 2097152, "depth": 1, "present": true, "zero": true, "data": false, "offset": OFFSET},
{ "start": 27262976, "length": 2097152, "depth": 1, "present": true, "zero": true, "data": false},
{ "start": 29360128, "length": 2097152, "depth": 1, "present": true, "zero": true, "data": false, "offset": OFFSET},
{ "start": 31457280, "length": 2097152, "depth": 1, "present": true, "zero": false, "data": true, "offset": OFFSET},
{ "start": 33554432, "length": 10485760, "depth": 1, "present": true, "zero": true, "data": false},
{ "start": 44040192, "length": 4194304, "depth": 0, "present": true, "zero": true, "data": false},
{ "start": 48234496, "length": 2097152, "depth": 1, "present": true, "zero": true, "data": false},
{ "start": 50331648, "length": 2097152, "depth": 1, "present": true, "zero": false, "data": true, "offset": OFFSET},
{ "start": 52428800, "length": 4194304, "depth": 0, "present": true, "zero": true, "data": false},
{ "start": 56623104, "length": 2097152, "depth": 1, "present": true, "zero": false, "data": true, "offset": OFFSET},
{ "start": 58720256, "length": 2097152, "depth": 1, "present": false, "zero": true, "data": false},
{ "start": 60817408, "length": 4194304, "depth": 0, "present": true, "zero": true, "data": false},
{ "start": 65011712, "length": 2097152, "depth": 1, "present": false, "zero": true, "data": false}]
iotests: Add test 179 to cover write zeroes with unmap No tests were covering write zeroes with unmap. Additionally, I needed to prove that my previous patches for correct status reporting and write zeroes optimizations actually had an impact. The test works for cluster_size between 8k and 2M (for smaller sizes, it fails because our allocation patterns are not contiguous with small clusters - in part, the largest consecutive allocation we tend to get is often bounded by the size covered by one L2 table). Note that testing for zero clusters is tricky: 'qemu-io map' reports whether data comes from the current layer of the image (useful for sniffing out which regions of the file have QCOW_OFLAG_ZERO) - but doesn't show which clusters have mappings; while 'qemu-img map' sees "zero":true for both unallocated and zero clusters for any qcow2 with no backing layer (so less useful at detecting true zero clusters), but reliably shows mappings. So we have to rely on both queries side-by-side at each point of the test. Signed-off-by: Eric Blake <eblake@redhat.com> Reviewed-by: Max Reitz <mreitz@redhat.com> Message-id: 20170507000552.20847-10-eblake@redhat.com Signed-off-by: Max Reitz <mreitz@redhat.com>
2017-05-07 02:05:49 +02:00
No errors were found on the image.
No errors were found on the image.
=== Testing cache optimization ===
wrote 2097152/2097152 bytes at offset 20971520
2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
wrote 2097152/2097152 bytes at offset 20971520
2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
wrote 2097152/2097152 bytes at offset 29360128
2 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
*** done