2014-02-21 16:24:08 +01:00
|
|
|
QA output created by 082
|
|
|
|
|
|
|
|
|
|
=== create: Options specified more than once ===
|
|
|
|
|
|
|
|
|
|
Testing: create -f foo -f qcow2 TEST_DIR/t.qcow2 128M
|
2020-07-10 18:13:13 +02:00
|
|
|
Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=134217728 lazy_refcounts=off refcount_bits=16
|
2014-09-30 21:31:27 +02:00
|
|
|
image: TEST_DIR/t.IMGFMT
|
|
|
|
|
file format: IMGFMT
|
2019-04-17 19:11:01 +02:00
|
|
|
virtual size: 128 MiB (134217728 bytes)
|
2014-02-21 16:24:08 +01:00
|
|
|
cluster_size: 65536
|
|
|
|
|
|
|
|
|
|
Testing: create -f qcow2 -o cluster_size=4k -o lazy_refcounts=on TEST_DIR/t.qcow2 128M
|
2020-07-10 18:13:13 +02:00
|
|
|
Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 cluster_size=4096 extended_l2=off compression_type=zlib size=134217728 lazy_refcounts=on refcount_bits=16
|
2014-11-27 17:28:01 +01:00
|
|
|
image: TEST_DIR/t.IMGFMT
|
|
|
|
|
file format: IMGFMT
|
2019-04-17 19:11:01 +02:00
|
|
|
virtual size: 128 MiB (134217728 bytes)
|
2014-02-21 16:24:08 +01:00
|
|
|
cluster_size: 4096
|
|
|
|
|
Format specific information:
|
|
|
|
|
compat: 1.1
|
qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206,
242, 255, 274, 280
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-05-07 10:25:18 +02:00
|
|
|
compression type: zlib
|
2014-02-21 16:24:08 +01:00
|
|
|
lazy refcounts: true
|
2015-02-10 21:28:44 +01:00
|
|
|
refcount bits: 16
|
2014-09-30 21:31:28 +02:00
|
|
|
corrupt: false
|
2020-07-10 18:13:13 +02:00
|
|
|
extended l2: false
|
2014-02-21 16:24:08 +01:00
|
|
|
|
|
|
|
|
Testing: create -f qcow2 -o cluster_size=4k -o lazy_refcounts=on -o cluster_size=8k TEST_DIR/t.qcow2 128M
|
2020-07-10 18:13:13 +02:00
|
|
|
Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 cluster_size=8192 extended_l2=off compression_type=zlib size=134217728 lazy_refcounts=on refcount_bits=16
|
2014-11-27 17:28:01 +01:00
|
|
|
image: TEST_DIR/t.IMGFMT
|
|
|
|
|
file format: IMGFMT
|
2019-04-17 19:11:01 +02:00
|
|
|
virtual size: 128 MiB (134217728 bytes)
|
2014-02-21 16:24:08 +01:00
|
|
|
cluster_size: 8192
|
|
|
|
|
Format specific information:
|
|
|
|
|
compat: 1.1
|
qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206,
242, 255, 274, 280
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-05-07 10:25:18 +02:00
|
|
|
compression type: zlib
|
2014-02-21 16:24:08 +01:00
|
|
|
lazy refcounts: true
|
2015-02-10 21:28:44 +01:00
|
|
|
refcount bits: 16
|
2014-09-30 21:31:28 +02:00
|
|
|
corrupt: false
|
2020-07-10 18:13:13 +02:00
|
|
|
extended l2: false
|
2014-02-21 16:24:08 +01:00
|
|
|
|
|
|
|
|
Testing: create -f qcow2 -o cluster_size=4k,cluster_size=8k TEST_DIR/t.qcow2 128M
|
2020-07-10 18:13:13 +02:00
|
|
|
Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 cluster_size=8192 extended_l2=off compression_type=zlib size=134217728 lazy_refcounts=off refcount_bits=16
|
2014-09-30 21:31:27 +02:00
|
|
|
image: TEST_DIR/t.IMGFMT
|
|
|
|
|
file format: IMGFMT
|
2019-04-17 19:11:01 +02:00
|
|
|
virtual size: 128 MiB (134217728 bytes)
|
2014-02-21 16:24:08 +01:00
|
|
|
cluster_size: 8192
|
|
|
|
|
|
|
|
|
|
=== create: help for -o ===
|
|
|
|
|
|
|
|
|
|
Testing: create -f qcow2 -o help TEST_DIR/t.qcow2 128M
|
|
|
|
|
Supported options:
|
2018-10-19 18:49:25 +02:00
|
|
|
backing_file=<str> - File name of a base image
|
|
|
|
|
backing_fmt=<str> - Image format of the base image
|
|
|
|
|
cluster_size=<size> - qcow2 cluster size
|
2019-07-08 20:47:03 +02:00
|
|
|
compat=<str> - Compatibility level (v2 [0.10] or v3 [1.1])
|
qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206,
242, 255, 274, 280
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-05-07 10:25:18 +02:00
|
|
|
compression_type=<str> - Compression method used for image cluster compression
|
2019-01-15 19:02:40 +01:00
|
|
|
data_file=<str> - File name of an external data file
|
2019-02-22 14:29:38 +01:00
|
|
|
data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
|
2018-10-19 18:49:25 +02:00
|
|
|
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
|
|
|
|
|
encrypt.cipher-mode=<str> - Name of encryption cipher mode
|
|
|
|
|
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
|
|
|
|
|
encrypt.hash-alg=<str> - Name of encryption hash algorithm
|
|
|
|
|
encrypt.iter-time=<num> - Time to spend in PBKDF in milliseconds
|
|
|
|
|
encrypt.ivgen-alg=<str> - Name of IV generator algorithm
|
|
|
|
|
encrypt.ivgen-hash-alg=<str> - Name of IV generator hash algorithm
|
|
|
|
|
encrypt.key-secret=<str> - ID of secret providing qcow AES key or LUKS passphrase
|
|
|
|
|
encryption=<bool (on/off)> - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes)
|
2020-07-10 18:13:13 +02:00
|
|
|
extended_l2=<bool (on/off)> - Extended L2 tables
|
file-posix: Mitigate file fragmentation with extent size hints
Especially when O_DIRECT is used with image files so that the page cache
indirection can't cause a merge of allocating requests, the file will
fragment on the file system layer, with a potentially very small
fragment size (this depends on the requests the guest sent).
On Linux, fragmentation can be reduced by setting an extent size hint
when creating the file (at least on XFS, it can't be set any more after
the first extent has been allocated), basically giving raw files a
"cluster size" for allocation.
This adds a create option to set the extent size hint, and changes the
default from not setting a hint to setting it to 1 MB. The main reason
why qcow2 defaults to smaller cluster sizes is that COW becomes more
expensive, which is not an issue with raw files, so we can choose a
larger size. The tradeoff here is only potentially wasted disk space.
For qcow2 (or other image formats) over file-posix, the advantage should
even be greater because they grow sequentially without leaving holes, so
there won't be wasted space. Setting even larger extent size hints for
such images may make sense. This can be done with the new option, but
let's keep the default conservative for now.
The effect is very visible with a test that intentionally creates a
badly fragmented file with qemu-img bench (the time difference while
creating the file is already remarkable) and then looks at the number of
extents and the time a simple "qemu-img map" takes.
Without an extent size hint:
$ ./qemu-img create -f raw -o extent_size_hint=0 ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=0
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 25.848 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 19.616 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 2000000 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m1,279s
user 0m0,043s
sys 0m1,226s
With the new default extent size hint of 1 MB:
$ ./qemu-img create -f raw -o extent_size_hint=1M ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=1048576
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 11.833 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 10.155 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 178 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m0,061s
user 0m0,040s
sys 0m0,014s
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200707142329.48303-1-kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-07 16:23:29 +02:00
|
|
|
extent_size_hint=<size> - Extent size hint for the image file, 0 to disable
|
2018-10-19 18:49:25 +02:00
|
|
|
lazy_refcounts=<bool (on/off)> - Postpone refcount updates
|
|
|
|
|
nocow=<bool (on/off)> - Turn off copy-on-write (valid only on btrfs)
|
|
|
|
|
preallocation=<str> - Preallocation mode (allowed values: off, metadata, falloc, full)
|
|
|
|
|
refcount_bits=<num> - Width of a reference count entry in bits
|
|
|
|
|
size=<size> - Virtual disk size
|
2014-02-21 16:24:08 +01:00
|
|
|
|
|
|
|
|
Testing: create -f qcow2 -o ? TEST_DIR/t.qcow2 128M
|
|
|
|
|
Supported options:
|
2018-10-19 18:49:25 +02:00
|
|
|
backing_file=<str> - File name of a base image
|
|
|
|
|
backing_fmt=<str> - Image format of the base image
|
|
|
|
|
cluster_size=<size> - qcow2 cluster size
|
2019-07-08 20:47:03 +02:00
|
|
|
compat=<str> - Compatibility level (v2 [0.10] or v3 [1.1])
|
qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206,
242, 255, 274, 280
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-05-07 10:25:18 +02:00
|
|
|
compression_type=<str> - Compression method used for image cluster compression
|
2019-01-15 19:02:40 +01:00
|
|
|
data_file=<str> - File name of an external data file
|
2019-02-22 14:29:38 +01:00
|
|
|
data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
|
2018-10-19 18:49:25 +02:00
|
|
|
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
|
|
|
|
|
encrypt.cipher-mode=<str> - Name of encryption cipher mode
|
|
|
|
|
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
|
|
|
|
|
encrypt.hash-alg=<str> - Name of encryption hash algorithm
|
|
|
|
|
encrypt.iter-time=<num> - Time to spend in PBKDF in milliseconds
|
|
|
|
|
encrypt.ivgen-alg=<str> - Name of IV generator algorithm
|
|
|
|
|
encrypt.ivgen-hash-alg=<str> - Name of IV generator hash algorithm
|
|
|
|
|
encrypt.key-secret=<str> - ID of secret providing qcow AES key or LUKS passphrase
|
|
|
|
|
encryption=<bool (on/off)> - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes)
|
2020-07-10 18:13:13 +02:00
|
|
|
extended_l2=<bool (on/off)> - Extended L2 tables
|
file-posix: Mitigate file fragmentation with extent size hints
Especially when O_DIRECT is used with image files so that the page cache
indirection can't cause a merge of allocating requests, the file will
fragment on the file system layer, with a potentially very small
fragment size (this depends on the requests the guest sent).
On Linux, fragmentation can be reduced by setting an extent size hint
when creating the file (at least on XFS, it can't be set any more after
the first extent has been allocated), basically giving raw files a
"cluster size" for allocation.
This adds a create option to set the extent size hint, and changes the
default from not setting a hint to setting it to 1 MB. The main reason
why qcow2 defaults to smaller cluster sizes is that COW becomes more
expensive, which is not an issue with raw files, so we can choose a
larger size. The tradeoff here is only potentially wasted disk space.
For qcow2 (or other image formats) over file-posix, the advantage should
even be greater because they grow sequentially without leaving holes, so
there won't be wasted space. Setting even larger extent size hints for
such images may make sense. This can be done with the new option, but
let's keep the default conservative for now.
The effect is very visible with a test that intentionally creates a
badly fragmented file with qemu-img bench (the time difference while
creating the file is already remarkable) and then looks at the number of
extents and the time a simple "qemu-img map" takes.
Without an extent size hint:
$ ./qemu-img create -f raw -o extent_size_hint=0 ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=0
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 25.848 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 19.616 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 2000000 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m1,279s
user 0m0,043s
sys 0m1,226s
With the new default extent size hint of 1 MB:
$ ./qemu-img create -f raw -o extent_size_hint=1M ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=1048576
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 11.833 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 10.155 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 178 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m0,061s
user 0m0,040s
sys 0m0,014s
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200707142329.48303-1-kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-07 16:23:29 +02:00
|
|
|
extent_size_hint=<size> - Extent size hint for the image file, 0 to disable
|
2018-10-19 18:49:25 +02:00
|
|
|
lazy_refcounts=<bool (on/off)> - Postpone refcount updates
|
|
|
|
|
nocow=<bool (on/off)> - Turn off copy-on-write (valid only on btrfs)
|
|
|
|
|
preallocation=<str> - Preallocation mode (allowed values: off, metadata, falloc, full)
|
|
|
|
|
refcount_bits=<num> - Width of a reference count entry in bits
|
|
|
|
|
size=<size> - Virtual disk size
|
2014-02-21 16:24:08 +01:00
|
|
|
|
|
|
|
|
Testing: create -f qcow2 -o cluster_size=4k,help TEST_DIR/t.qcow2 128M
|
|
|
|
|
Supported options:
|
2018-10-19 18:49:25 +02:00
|
|
|
backing_file=<str> - File name of a base image
|
|
|
|
|
backing_fmt=<str> - Image format of the base image
|
|
|
|
|
cluster_size=<size> - qcow2 cluster size
|
2019-07-08 20:47:03 +02:00
|
|
|
compat=<str> - Compatibility level (v2 [0.10] or v3 [1.1])
|
qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206,
242, 255, 274, 280
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-05-07 10:25:18 +02:00
|
|
|
compression_type=<str> - Compression method used for image cluster compression
|
2019-01-15 19:02:40 +01:00
|
|
|
data_file=<str> - File name of an external data file
|
2019-02-22 14:29:38 +01:00
|
|
|
data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
|
2018-10-19 18:49:25 +02:00
|
|
|
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
|
|
|
|
|
encrypt.cipher-mode=<str> - Name of encryption cipher mode
|
|
|
|
|
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
|
|
|
|
|
encrypt.hash-alg=<str> - Name of encryption hash algorithm
|
|
|
|
|
encrypt.iter-time=<num> - Time to spend in PBKDF in milliseconds
|
|
|
|
|
encrypt.ivgen-alg=<str> - Name of IV generator algorithm
|
|
|
|
|
encrypt.ivgen-hash-alg=<str> - Name of IV generator hash algorithm
|
|
|
|
|
encrypt.key-secret=<str> - ID of secret providing qcow AES key or LUKS passphrase
|
|
|
|
|
encryption=<bool (on/off)> - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes)
|
2020-07-10 18:13:13 +02:00
|
|
|
extended_l2=<bool (on/off)> - Extended L2 tables
|
file-posix: Mitigate file fragmentation with extent size hints
Especially when O_DIRECT is used with image files so that the page cache
indirection can't cause a merge of allocating requests, the file will
fragment on the file system layer, with a potentially very small
fragment size (this depends on the requests the guest sent).
On Linux, fragmentation can be reduced by setting an extent size hint
when creating the file (at least on XFS, it can't be set any more after
the first extent has been allocated), basically giving raw files a
"cluster size" for allocation.
This adds a create option to set the extent size hint, and changes the
default from not setting a hint to setting it to 1 MB. The main reason
why qcow2 defaults to smaller cluster sizes is that COW becomes more
expensive, which is not an issue with raw files, so we can choose a
larger size. The tradeoff here is only potentially wasted disk space.
For qcow2 (or other image formats) over file-posix, the advantage should
even be greater because they grow sequentially without leaving holes, so
there won't be wasted space. Setting even larger extent size hints for
such images may make sense. This can be done with the new option, but
let's keep the default conservative for now.
The effect is very visible with a test that intentionally creates a
badly fragmented file with qemu-img bench (the time difference while
creating the file is already remarkable) and then looks at the number of
extents and the time a simple "qemu-img map" takes.
Without an extent size hint:
$ ./qemu-img create -f raw -o extent_size_hint=0 ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=0
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 25.848 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 19.616 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 2000000 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m1,279s
user 0m0,043s
sys 0m1,226s
With the new default extent size hint of 1 MB:
$ ./qemu-img create -f raw -o extent_size_hint=1M ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=1048576
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 11.833 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 10.155 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 178 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m0,061s
user 0m0,040s
sys 0m0,014s
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200707142329.48303-1-kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-07 16:23:29 +02:00
|
|
|
extent_size_hint=<size> - Extent size hint for the image file, 0 to disable
|
2018-10-19 18:49:25 +02:00
|
|
|
lazy_refcounts=<bool (on/off)> - Postpone refcount updates
|
|
|
|
|
nocow=<bool (on/off)> - Turn off copy-on-write (valid only on btrfs)
|
|
|
|
|
preallocation=<str> - Preallocation mode (allowed values: off, metadata, falloc, full)
|
|
|
|
|
refcount_bits=<num> - Width of a reference count entry in bits
|
|
|
|
|
size=<size> - Virtual disk size
|
2014-02-21 16:24:08 +01:00
|
|
|
|
|
|
|
|
Testing: create -f qcow2 -o cluster_size=4k,? TEST_DIR/t.qcow2 128M
|
|
|
|
|
Supported options:
|
2018-10-19 18:49:25 +02:00
|
|
|
backing_file=<str> - File name of a base image
|
|
|
|
|
backing_fmt=<str> - Image format of the base image
|
|
|
|
|
cluster_size=<size> - qcow2 cluster size
|
2019-07-08 20:47:03 +02:00
|
|
|
compat=<str> - Compatibility level (v2 [0.10] or v3 [1.1])
|
qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206,
242, 255, 274, 280
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-05-07 10:25:18 +02:00
|
|
|
compression_type=<str> - Compression method used for image cluster compression
|
2019-01-15 19:02:40 +01:00
|
|
|
data_file=<str> - File name of an external data file
|
2019-02-22 14:29:38 +01:00
|
|
|
data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
|
2018-10-19 18:49:25 +02:00
|
|
|
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
|
|
|
|
|
encrypt.cipher-mode=<str> - Name of encryption cipher mode
|
|
|
|
|
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
|
|
|
|
|
encrypt.hash-alg=<str> - Name of encryption hash algorithm
|
|
|
|
|
encrypt.iter-time=<num> - Time to spend in PBKDF in milliseconds
|
|
|
|
|
encrypt.ivgen-alg=<str> - Name of IV generator algorithm
|
|
|
|
|
encrypt.ivgen-hash-alg=<str> - Name of IV generator hash algorithm
|
|
|
|
|
encrypt.key-secret=<str> - ID of secret providing qcow AES key or LUKS passphrase
|
|
|
|
|
encryption=<bool (on/off)> - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes)
|
2020-07-10 18:13:13 +02:00
|
|
|
extended_l2=<bool (on/off)> - Extended L2 tables
|
file-posix: Mitigate file fragmentation with extent size hints
Especially when O_DIRECT is used with image files so that the page cache
indirection can't cause a merge of allocating requests, the file will
fragment on the file system layer, with a potentially very small
fragment size (this depends on the requests the guest sent).
On Linux, fragmentation can be reduced by setting an extent size hint
when creating the file (at least on XFS, it can't be set any more after
the first extent has been allocated), basically giving raw files a
"cluster size" for allocation.
This adds a create option to set the extent size hint, and changes the
default from not setting a hint to setting it to 1 MB. The main reason
why qcow2 defaults to smaller cluster sizes is that COW becomes more
expensive, which is not an issue with raw files, so we can choose a
larger size. The tradeoff here is only potentially wasted disk space.
For qcow2 (or other image formats) over file-posix, the advantage should
even be greater because they grow sequentially without leaving holes, so
there won't be wasted space. Setting even larger extent size hints for
such images may make sense. This can be done with the new option, but
let's keep the default conservative for now.
The effect is very visible with a test that intentionally creates a
badly fragmented file with qemu-img bench (the time difference while
creating the file is already remarkable) and then looks at the number of
extents and the time a simple "qemu-img map" takes.
Without an extent size hint:
$ ./qemu-img create -f raw -o extent_size_hint=0 ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=0
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 25.848 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 19.616 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 2000000 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m1,279s
user 0m0,043s
sys 0m1,226s
With the new default extent size hint of 1 MB:
$ ./qemu-img create -f raw -o extent_size_hint=1M ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=1048576
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 11.833 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 10.155 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 178 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m0,061s
user 0m0,040s
sys 0m0,014s
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200707142329.48303-1-kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-07 16:23:29 +02:00
|
|
|
extent_size_hint=<size> - Extent size hint for the image file, 0 to disable
|
2018-10-19 18:49:25 +02:00
|
|
|
lazy_refcounts=<bool (on/off)> - Postpone refcount updates
|
|
|
|
|
nocow=<bool (on/off)> - Turn off copy-on-write (valid only on btrfs)
|
|
|
|
|
preallocation=<str> - Preallocation mode (allowed values: off, metadata, falloc, full)
|
|
|
|
|
refcount_bits=<num> - Width of a reference count entry in bits
|
|
|
|
|
size=<size> - Virtual disk size
|
2014-02-21 16:24:08 +01:00
|
|
|
|
|
|
|
|
Testing: create -f qcow2 -o help,cluster_size=4k TEST_DIR/t.qcow2 128M
|
|
|
|
|
Supported options:
|
2018-10-19 18:49:25 +02:00
|
|
|
backing_file=<str> - File name of a base image
|
|
|
|
|
backing_fmt=<str> - Image format of the base image
|
|
|
|
|
cluster_size=<size> - qcow2 cluster size
|
2019-07-08 20:47:03 +02:00
|
|
|
compat=<str> - Compatibility level (v2 [0.10] or v3 [1.1])
|
qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206,
242, 255, 274, 280
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-05-07 10:25:18 +02:00
|
|
|
compression_type=<str> - Compression method used for image cluster compression
|
2019-01-15 19:02:40 +01:00
|
|
|
data_file=<str> - File name of an external data file
|
2019-02-22 14:29:38 +01:00
|
|
|
data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
|
2018-10-19 18:49:25 +02:00
|
|
|
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
|
|
|
|
|
encrypt.cipher-mode=<str> - Name of encryption cipher mode
|
|
|
|
|
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
|
|
|
|
|
encrypt.hash-alg=<str> - Name of encryption hash algorithm
|
|
|
|
|
encrypt.iter-time=<num> - Time to spend in PBKDF in milliseconds
|
|
|
|
|
encrypt.ivgen-alg=<str> - Name of IV generator algorithm
|
|
|
|
|
encrypt.ivgen-hash-alg=<str> - Name of IV generator hash algorithm
|
|
|
|
|
encrypt.key-secret=<str> - ID of secret providing qcow AES key or LUKS passphrase
|
|
|
|
|
encryption=<bool (on/off)> - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes)
|
2020-07-10 18:13:13 +02:00
|
|
|
extended_l2=<bool (on/off)> - Extended L2 tables
|
file-posix: Mitigate file fragmentation with extent size hints
Especially when O_DIRECT is used with image files so that the page cache
indirection can't cause a merge of allocating requests, the file will
fragment on the file system layer, with a potentially very small
fragment size (this depends on the requests the guest sent).
On Linux, fragmentation can be reduced by setting an extent size hint
when creating the file (at least on XFS, it can't be set any more after
the first extent has been allocated), basically giving raw files a
"cluster size" for allocation.
This adds a create option to set the extent size hint, and changes the
default from not setting a hint to setting it to 1 MB. The main reason
why qcow2 defaults to smaller cluster sizes is that COW becomes more
expensive, which is not an issue with raw files, so we can choose a
larger size. The tradeoff here is only potentially wasted disk space.
For qcow2 (or other image formats) over file-posix, the advantage should
even be greater because they grow sequentially without leaving holes, so
there won't be wasted space. Setting even larger extent size hints for
such images may make sense. This can be done with the new option, but
let's keep the default conservative for now.
The effect is very visible with a test that intentionally creates a
badly fragmented file with qemu-img bench (the time difference while
creating the file is already remarkable) and then looks at the number of
extents and the time a simple "qemu-img map" takes.
Without an extent size hint:
$ ./qemu-img create -f raw -o extent_size_hint=0 ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=0
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 25.848 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 19.616 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 2000000 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m1,279s
user 0m0,043s
sys 0m1,226s
With the new default extent size hint of 1 MB:
$ ./qemu-img create -f raw -o extent_size_hint=1M ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=1048576
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 11.833 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 10.155 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 178 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m0,061s
user 0m0,040s
sys 0m0,014s
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200707142329.48303-1-kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-07 16:23:29 +02:00
|
|
|
extent_size_hint=<size> - Extent size hint for the image file, 0 to disable
|
2018-10-19 18:49:25 +02:00
|
|
|
lazy_refcounts=<bool (on/off)> - Postpone refcount updates
|
|
|
|
|
nocow=<bool (on/off)> - Turn off copy-on-write (valid only on btrfs)
|
|
|
|
|
preallocation=<str> - Preallocation mode (allowed values: off, metadata, falloc, full)
|
|
|
|
|
refcount_bits=<num> - Width of a reference count entry in bits
|
|
|
|
|
size=<size> - Virtual disk size
|
2014-02-21 16:24:08 +01:00
|
|
|
|
|
|
|
|
Testing: create -f qcow2 -o ?,cluster_size=4k TEST_DIR/t.qcow2 128M
|
|
|
|
|
Supported options:
|
2018-10-19 18:49:25 +02:00
|
|
|
backing_file=<str> - File name of a base image
|
|
|
|
|
backing_fmt=<str> - Image format of the base image
|
|
|
|
|
cluster_size=<size> - qcow2 cluster size
|
2019-07-08 20:47:03 +02:00
|
|
|
compat=<str> - Compatibility level (v2 [0.10] or v3 [1.1])
|
qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206,
242, 255, 274, 280
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-05-07 10:25:18 +02:00
|
|
|
compression_type=<str> - Compression method used for image cluster compression
|
2019-01-15 19:02:40 +01:00
|
|
|
data_file=<str> - File name of an external data file
|
2019-02-22 14:29:38 +01:00
|
|
|
data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
|
2018-10-19 18:49:25 +02:00
|
|
|
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
|
|
|
|
|
encrypt.cipher-mode=<str> - Name of encryption cipher mode
|
|
|
|
|
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
|
|
|
|
|
encrypt.hash-alg=<str> - Name of encryption hash algorithm
|
|
|
|
|
encrypt.iter-time=<num> - Time to spend in PBKDF in milliseconds
|
|
|
|
|
encrypt.ivgen-alg=<str> - Name of IV generator algorithm
|
|
|
|
|
encrypt.ivgen-hash-alg=<str> - Name of IV generator hash algorithm
|
|
|
|
|
encrypt.key-secret=<str> - ID of secret providing qcow AES key or LUKS passphrase
|
|
|
|
|
encryption=<bool (on/off)> - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes)
|
2020-07-10 18:13:13 +02:00
|
|
|
extended_l2=<bool (on/off)> - Extended L2 tables
|
file-posix: Mitigate file fragmentation with extent size hints
Especially when O_DIRECT is used with image files so that the page cache
indirection can't cause a merge of allocating requests, the file will
fragment on the file system layer, with a potentially very small
fragment size (this depends on the requests the guest sent).
On Linux, fragmentation can be reduced by setting an extent size hint
when creating the file (at least on XFS, it can't be set any more after
the first extent has been allocated), basically giving raw files a
"cluster size" for allocation.
This adds a create option to set the extent size hint, and changes the
default from not setting a hint to setting it to 1 MB. The main reason
why qcow2 defaults to smaller cluster sizes is that COW becomes more
expensive, which is not an issue with raw files, so we can choose a
larger size. The tradeoff here is only potentially wasted disk space.
For qcow2 (or other image formats) over file-posix, the advantage should
even be greater because they grow sequentially without leaving holes, so
there won't be wasted space. Setting even larger extent size hints for
such images may make sense. This can be done with the new option, but
let's keep the default conservative for now.
The effect is very visible with a test that intentionally creates a
badly fragmented file with qemu-img bench (the time difference while
creating the file is already remarkable) and then looks at the number of
extents and the time a simple "qemu-img map" takes.
Without an extent size hint:
$ ./qemu-img create -f raw -o extent_size_hint=0 ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=0
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 25.848 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 19.616 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 2000000 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m1,279s
user 0m0,043s
sys 0m1,226s
With the new default extent size hint of 1 MB:
$ ./qemu-img create -f raw -o extent_size_hint=1M ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=1048576
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 11.833 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 10.155 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 178 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m0,061s
user 0m0,040s
sys 0m0,014s
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200707142329.48303-1-kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-07 16:23:29 +02:00
|
|
|
extent_size_hint=<size> - Extent size hint for the image file, 0 to disable
|
2018-10-19 18:49:25 +02:00
|
|
|
lazy_refcounts=<bool (on/off)> - Postpone refcount updates
|
|
|
|
|
nocow=<bool (on/off)> - Turn off copy-on-write (valid only on btrfs)
|
|
|
|
|
preallocation=<str> - Preallocation mode (allowed values: off, metadata, falloc, full)
|
|
|
|
|
refcount_bits=<num> - Width of a reference count entry in bits
|
|
|
|
|
size=<size> - Virtual disk size
|
2014-02-21 16:24:08 +01:00
|
|
|
|
|
|
|
|
Testing: create -f qcow2 -o cluster_size=4k -o help TEST_DIR/t.qcow2 128M
|
|
|
|
|
Supported options:
|
2018-10-19 18:49:25 +02:00
|
|
|
backing_file=<str> - File name of a base image
|
|
|
|
|
backing_fmt=<str> - Image format of the base image
|
|
|
|
|
cluster_size=<size> - qcow2 cluster size
|
2019-07-08 20:47:03 +02:00
|
|
|
compat=<str> - Compatibility level (v2 [0.10] or v3 [1.1])
|
qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206,
242, 255, 274, 280
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-05-07 10:25:18 +02:00
|
|
|
compression_type=<str> - Compression method used for image cluster compression
|
2019-01-15 19:02:40 +01:00
|
|
|
data_file=<str> - File name of an external data file
|
2019-02-22 14:29:38 +01:00
|
|
|
data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
|
2018-10-19 18:49:25 +02:00
|
|
|
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
|
|
|
|
|
encrypt.cipher-mode=<str> - Name of encryption cipher mode
|
|
|
|
|
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
|
|
|
|
|
encrypt.hash-alg=<str> - Name of encryption hash algorithm
|
|
|
|
|
encrypt.iter-time=<num> - Time to spend in PBKDF in milliseconds
|
|
|
|
|
encrypt.ivgen-alg=<str> - Name of IV generator algorithm
|
|
|
|
|
encrypt.ivgen-hash-alg=<str> - Name of IV generator hash algorithm
|
|
|
|
|
encrypt.key-secret=<str> - ID of secret providing qcow AES key or LUKS passphrase
|
|
|
|
|
encryption=<bool (on/off)> - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes)
|
2020-07-10 18:13:13 +02:00
|
|
|
extended_l2=<bool (on/off)> - Extended L2 tables
|
file-posix: Mitigate file fragmentation with extent size hints
Especially when O_DIRECT is used with image files so that the page cache
indirection can't cause a merge of allocating requests, the file will
fragment on the file system layer, with a potentially very small
fragment size (this depends on the requests the guest sent).
On Linux, fragmentation can be reduced by setting an extent size hint
when creating the file (at least on XFS, it can't be set any more after
the first extent has been allocated), basically giving raw files a
"cluster size" for allocation.
This adds a create option to set the extent size hint, and changes the
default from not setting a hint to setting it to 1 MB. The main reason
why qcow2 defaults to smaller cluster sizes is that COW becomes more
expensive, which is not an issue with raw files, so we can choose a
larger size. The tradeoff here is only potentially wasted disk space.
For qcow2 (or other image formats) over file-posix, the advantage should
even be greater because they grow sequentially without leaving holes, so
there won't be wasted space. Setting even larger extent size hints for
such images may make sense. This can be done with the new option, but
let's keep the default conservative for now.
The effect is very visible with a test that intentionally creates a
badly fragmented file with qemu-img bench (the time difference while
creating the file is already remarkable) and then looks at the number of
extents and the time a simple "qemu-img map" takes.
Without an extent size hint:
$ ./qemu-img create -f raw -o extent_size_hint=0 ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=0
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 25.848 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 19.616 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 2000000 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m1,279s
user 0m0,043s
sys 0m1,226s
With the new default extent size hint of 1 MB:
$ ./qemu-img create -f raw -o extent_size_hint=1M ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=1048576
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 11.833 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 10.155 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 178 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m0,061s
user 0m0,040s
sys 0m0,014s
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200707142329.48303-1-kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-07 16:23:29 +02:00
|
|
|
extent_size_hint=<size> - Extent size hint for the image file, 0 to disable
|
2018-10-19 18:49:25 +02:00
|
|
|
lazy_refcounts=<bool (on/off)> - Postpone refcount updates
|
|
|
|
|
nocow=<bool (on/off)> - Turn off copy-on-write (valid only on btrfs)
|
|
|
|
|
preallocation=<str> - Preallocation mode (allowed values: off, metadata, falloc, full)
|
|
|
|
|
refcount_bits=<num> - Width of a reference count entry in bits
|
|
|
|
|
size=<size> - Virtual disk size
|
2014-02-21 16:24:08 +01:00
|
|
|
|
|
|
|
|
Testing: create -f qcow2 -o cluster_size=4k -o ? TEST_DIR/t.qcow2 128M
|
|
|
|
|
Supported options:
|
2018-10-19 18:49:25 +02:00
|
|
|
backing_file=<str> - File name of a base image
|
|
|
|
|
backing_fmt=<str> - Image format of the base image
|
|
|
|
|
cluster_size=<size> - qcow2 cluster size
|
2019-07-08 20:47:03 +02:00
|
|
|
compat=<str> - Compatibility level (v2 [0.10] or v3 [1.1])
|
qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206,
242, 255, 274, 280
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-05-07 10:25:18 +02:00
|
|
|
compression_type=<str> - Compression method used for image cluster compression
|
2019-01-15 19:02:40 +01:00
|
|
|
data_file=<str> - File name of an external data file
|
2019-02-22 14:29:38 +01:00
|
|
|
data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
|
2018-10-19 18:49:25 +02:00
|
|
|
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
|
|
|
|
|
encrypt.cipher-mode=<str> - Name of encryption cipher mode
|
|
|
|
|
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
|
|
|
|
|
encrypt.hash-alg=<str> - Name of encryption hash algorithm
|
|
|
|
|
encrypt.iter-time=<num> - Time to spend in PBKDF in milliseconds
|
|
|
|
|
encrypt.ivgen-alg=<str> - Name of IV generator algorithm
|
|
|
|
|
encrypt.ivgen-hash-alg=<str> - Name of IV generator hash algorithm
|
|
|
|
|
encrypt.key-secret=<str> - ID of secret providing qcow AES key or LUKS passphrase
|
|
|
|
|
encryption=<bool (on/off)> - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes)
|
2020-07-10 18:13:13 +02:00
|
|
|
extended_l2=<bool (on/off)> - Extended L2 tables
|
file-posix: Mitigate file fragmentation with extent size hints
Especially when O_DIRECT is used with image files so that the page cache
indirection can't cause a merge of allocating requests, the file will
fragment on the file system layer, with a potentially very small
fragment size (this depends on the requests the guest sent).
On Linux, fragmentation can be reduced by setting an extent size hint
when creating the file (at least on XFS, it can't be set any more after
the first extent has been allocated), basically giving raw files a
"cluster size" for allocation.
This adds a create option to set the extent size hint, and changes the
default from not setting a hint to setting it to 1 MB. The main reason
why qcow2 defaults to smaller cluster sizes is that COW becomes more
expensive, which is not an issue with raw files, so we can choose a
larger size. The tradeoff here is only potentially wasted disk space.
For qcow2 (or other image formats) over file-posix, the advantage should
even be greater because they grow sequentially without leaving holes, so
there won't be wasted space. Setting even larger extent size hints for
such images may make sense. This can be done with the new option, but
let's keep the default conservative for now.
The effect is very visible with a test that intentionally creates a
badly fragmented file with qemu-img bench (the time difference while
creating the file is already remarkable) and then looks at the number of
extents and the time a simple "qemu-img map" takes.
Without an extent size hint:
$ ./qemu-img create -f raw -o extent_size_hint=0 ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=0
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 25.848 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 19.616 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 2000000 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m1,279s
user 0m0,043s
sys 0m1,226s
With the new default extent size hint of 1 MB:
$ ./qemu-img create -f raw -o extent_size_hint=1M ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=1048576
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 11.833 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 10.155 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 178 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m0,061s
user 0m0,040s
sys 0m0,014s
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200707142329.48303-1-kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-07 16:23:29 +02:00
|
|
|
extent_size_hint=<size> - Extent size hint for the image file, 0 to disable
|
2018-10-19 18:49:25 +02:00
|
|
|
lazy_refcounts=<bool (on/off)> - Postpone refcount updates
|
|
|
|
|
nocow=<bool (on/off)> - Turn off copy-on-write (valid only on btrfs)
|
|
|
|
|
preallocation=<str> - Preallocation mode (allowed values: off, metadata, falloc, full)
|
|
|
|
|
refcount_bits=<num> - Width of a reference count entry in bits
|
|
|
|
|
size=<size> - Virtual disk size
|
2014-02-21 16:24:08 +01:00
|
|
|
|
iotests: Specify explicit backing format where sensible
There are many existing qcow2 images that specify a backing file but
no format. This has been the source of CVEs in the past, but has
become more prominent of a problem now that libvirt has switched to
-blockdev. With older -drive, at least the probing was always done by
qemu (so the only risk of a changed format between successive boots of
a guest was if qemu was upgraded and probed differently). But with
newer -blockdev, libvirt must specify a format; if libvirt guesses raw
where the image was formatted, this results in data corruption visible
to the guest; conversely, if libvirt guesses qcow2 where qemu was
using raw, this can result in potential security holes, so modern
libvirt instead refuses to use images without explicit backing format.
The change in libvirt to reject images without explicit backing format
has pointed out that a number of tools have been far too reliant on
probing in the past. It's time to set a better example in our own
iotests of properly setting this parameter.
iotest calls to create, rebase, and convert are all impacted to some
degree. It's a bit annoying that we are inconsistent on command line
- while all of those accept -o backing_file=...,backing_fmt=..., the
shortcuts are different: create and rebase have -b and -F, while
convert has -B but no -F. (amend has no shortcuts, but the previous
patch just deprecated the use of amend to change backing chains).
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-9-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-06 22:39:52 +02:00
|
|
|
Testing: create -f qcow2 -u -o backing_file=TEST_DIR/t.qcow2,,help -F qcow2 TEST_DIR/t.qcow2 128M
|
2020-07-10 18:13:13 +02:00
|
|
|
Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=134217728 backing_file=TEST_DIR/t.qcow2,,help backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
|
2014-02-21 16:24:08 +01:00
|
|
|
|
iotests: Specify explicit backing format where sensible
There are many existing qcow2 images that specify a backing file but
no format. This has been the source of CVEs in the past, but has
become more prominent of a problem now that libvirt has switched to
-blockdev. With older -drive, at least the probing was always done by
qemu (so the only risk of a changed format between successive boots of
a guest was if qemu was upgraded and probed differently). But with
newer -blockdev, libvirt must specify a format; if libvirt guesses raw
where the image was formatted, this results in data corruption visible
to the guest; conversely, if libvirt guesses qcow2 where qemu was
using raw, this can result in potential security holes, so modern
libvirt instead refuses to use images without explicit backing format.
The change in libvirt to reject images without explicit backing format
has pointed out that a number of tools have been far too reliant on
probing in the past. It's time to set a better example in our own
iotests of properly setting this parameter.
iotest calls to create, rebase, and convert are all impacted to some
degree. It's a bit annoying that we are inconsistent on command line
- while all of those accept -o backing_file=...,backing_fmt=..., the
shortcuts are different: create and rebase have -b and -F, while
convert has -B but no -F. (amend has no shortcuts, but the previous
patch just deprecated the use of amend to change backing chains).
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-9-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-06 22:39:52 +02:00
|
|
|
Testing: create -f qcow2 -u -o backing_file=TEST_DIR/t.qcow2,,? -F qcow2 TEST_DIR/t.qcow2 128M
|
2020-07-10 18:13:13 +02:00
|
|
|
Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=134217728 backing_file=TEST_DIR/t.qcow2,,? backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16
|
2014-02-21 16:24:08 +01:00
|
|
|
|
|
|
|
|
Testing: create -f qcow2 -o backing_file=TEST_DIR/t.qcow2, -o help TEST_DIR/t.qcow2 128M
|
|
|
|
|
qemu-img: Invalid option list: backing_file=TEST_DIR/t.qcow2,
|
|
|
|
|
|
|
|
|
|
Testing: create -f qcow2 -o backing_file=TEST_DIR/t.qcow2 -o ,help TEST_DIR/t.qcow2 128M
|
|
|
|
|
qemu-img: Invalid option list: ,help
|
|
|
|
|
|
|
|
|
|
Testing: create -f qcow2 -o backing_file=TEST_DIR/t.qcow2 -o ,, -o help TEST_DIR/t.qcow2 128M
|
|
|
|
|
qemu-img: Invalid option list: ,,
|
|
|
|
|
|
|
|
|
|
Testing: create -f qcow2 -o help
|
2019-04-13 17:20:37 +02:00
|
|
|
Supported qcow2 options:
|
2018-10-19 18:49:25 +02:00
|
|
|
backing_file=<str> - File name of a base image
|
|
|
|
|
backing_fmt=<str> - Image format of the base image
|
|
|
|
|
cluster_size=<size> - qcow2 cluster size
|
2019-07-08 20:47:03 +02:00
|
|
|
compat=<str> - Compatibility level (v2 [0.10] or v3 [1.1])
|
qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206,
242, 255, 274, 280
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-05-07 10:25:18 +02:00
|
|
|
compression_type=<str> - Compression method used for image cluster compression
|
2019-01-15 19:02:40 +01:00
|
|
|
data_file=<str> - File name of an external data file
|
2019-02-22 14:29:38 +01:00
|
|
|
data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
|
2018-10-19 18:49:25 +02:00
|
|
|
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
|
|
|
|
|
encrypt.cipher-mode=<str> - Name of encryption cipher mode
|
|
|
|
|
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
|
|
|
|
|
encrypt.hash-alg=<str> - Name of encryption hash algorithm
|
|
|
|
|
encrypt.iter-time=<num> - Time to spend in PBKDF in milliseconds
|
|
|
|
|
encrypt.ivgen-alg=<str> - Name of IV generator algorithm
|
|
|
|
|
encrypt.ivgen-hash-alg=<str> - Name of IV generator hash algorithm
|
|
|
|
|
encrypt.key-secret=<str> - ID of secret providing qcow AES key or LUKS passphrase
|
|
|
|
|
encryption=<bool (on/off)> - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes)
|
2020-07-10 18:13:13 +02:00
|
|
|
extended_l2=<bool (on/off)> - Extended L2 tables
|
2018-10-19 18:49:25 +02:00
|
|
|
lazy_refcounts=<bool (on/off)> - Postpone refcount updates
|
|
|
|
|
preallocation=<str> - Preallocation mode (allowed values: off, metadata, falloc, full)
|
|
|
|
|
refcount_bits=<num> - Width of a reference count entry in bits
|
|
|
|
|
size=<size> - Virtual disk size
|
2014-02-21 16:24:08 +01:00
|
|
|
|
2019-04-13 17:20:37 +02:00
|
|
|
The protocol level may support further options.
|
|
|
|
|
Specify the target filename to include those options.
|
|
|
|
|
|
2014-02-21 16:24:08 +01:00
|
|
|
Testing: create -o help
|
2019-04-13 17:20:37 +02:00
|
|
|
Supported raw options:
|
2018-10-19 18:49:25 +02:00
|
|
|
size=<size> - Virtual disk size
|
2014-02-21 16:24:08 +01:00
|
|
|
|
2019-04-13 17:20:37 +02:00
|
|
|
The protocol level may support further options.
|
|
|
|
|
Specify the target filename to include those options.
|
|
|
|
|
|
2018-05-09 23:00:22 +02:00
|
|
|
Testing: create -f bochs -o help
|
|
|
|
|
qemu-img: Format driver 'bochs' does not support image creation
|
|
|
|
|
|
2014-02-21 16:24:08 +01:00
|
|
|
=== convert: Options specified more than once ===
|
|
|
|
|
|
|
|
|
|
Testing: create -f qcow2 TEST_DIR/t.qcow2 128M
|
2020-07-10 18:13:13 +02:00
|
|
|
Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=134217728 lazy_refcounts=off refcount_bits=16
|
2014-02-21 16:24:08 +01:00
|
|
|
|
|
|
|
|
Testing: convert -f foo -f qcow2 TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.base
|
2014-09-30 21:31:27 +02:00
|
|
|
image: TEST_DIR/t.IMGFMT.base
|
2014-02-21 16:24:08 +01:00
|
|
|
file format: raw
|
2019-04-17 19:11:01 +02:00
|
|
|
virtual size: 128 MiB (134217728 bytes)
|
2014-02-21 16:24:08 +01:00
|
|
|
|
|
|
|
|
Testing: convert -O foo -O qcow2 TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.base
|
2014-09-30 21:31:27 +02:00
|
|
|
image: TEST_DIR/t.IMGFMT.base
|
|
|
|
|
file format: IMGFMT
|
2019-04-17 19:11:01 +02:00
|
|
|
virtual size: 128 MiB (134217728 bytes)
|
2014-02-21 16:24:08 +01:00
|
|
|
cluster_size: 65536
|
|
|
|
|
|
|
|
|
|
Testing: convert -O qcow2 -o cluster_size=4k -o lazy_refcounts=on TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.base
|
2014-11-27 17:28:01 +01:00
|
|
|
image: TEST_DIR/t.IMGFMT.base
|
|
|
|
|
file format: IMGFMT
|
2019-04-17 19:11:01 +02:00
|
|
|
virtual size: 128 MiB (134217728 bytes)
|
2014-02-21 16:24:08 +01:00
|
|
|
cluster_size: 4096
|
|
|
|
|
Format specific information:
|
|
|
|
|
compat: 1.1
|
qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206,
242, 255, 274, 280
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-05-07 10:25:18 +02:00
|
|
|
compression type: zlib
|
2014-02-21 16:24:08 +01:00
|
|
|
lazy refcounts: true
|
2015-02-10 21:28:44 +01:00
|
|
|
refcount bits: 16
|
2014-09-30 21:31:28 +02:00
|
|
|
corrupt: false
|
2020-07-10 18:13:13 +02:00
|
|
|
extended l2: false
|
2014-02-21 16:24:08 +01:00
|
|
|
|
|
|
|
|
Testing: convert -O qcow2 -o cluster_size=4k -o lazy_refcounts=on -o cluster_size=8k TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.base
|
2014-11-27 17:28:01 +01:00
|
|
|
image: TEST_DIR/t.IMGFMT.base
|
|
|
|
|
file format: IMGFMT
|
2019-04-17 19:11:01 +02:00
|
|
|
virtual size: 128 MiB (134217728 bytes)
|
2014-02-21 16:24:08 +01:00
|
|
|
cluster_size: 8192
|
|
|
|
|
Format specific information:
|
|
|
|
|
compat: 1.1
|
qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206,
242, 255, 274, 280
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-05-07 10:25:18 +02:00
|
|
|
compression type: zlib
|
2014-02-21 16:24:08 +01:00
|
|
|
lazy refcounts: true
|
2015-02-10 21:28:44 +01:00
|
|
|
refcount bits: 16
|
2014-09-30 21:31:28 +02:00
|
|
|
corrupt: false
|
2020-07-10 18:13:13 +02:00
|
|
|
extended l2: false
|
2014-02-21 16:24:08 +01:00
|
|
|
|
|
|
|
|
Testing: convert -O qcow2 -o cluster_size=4k,cluster_size=8k TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.base
|
2014-09-30 21:31:27 +02:00
|
|
|
image: TEST_DIR/t.IMGFMT.base
|
|
|
|
|
file format: IMGFMT
|
2019-04-17 19:11:01 +02:00
|
|
|
virtual size: 128 MiB (134217728 bytes)
|
2014-02-21 16:24:08 +01:00
|
|
|
cluster_size: 8192
|
|
|
|
|
|
|
|
|
|
=== convert: help for -o ===
|
|
|
|
|
|
|
|
|
|
Testing: convert -O qcow2 -o help TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.base
|
|
|
|
|
Supported options:
|
2018-10-19 18:49:25 +02:00
|
|
|
backing_file=<str> - File name of a base image
|
|
|
|
|
backing_fmt=<str> - Image format of the base image
|
|
|
|
|
cluster_size=<size> - qcow2 cluster size
|
2019-07-08 20:47:03 +02:00
|
|
|
compat=<str> - Compatibility level (v2 [0.10] or v3 [1.1])
|
qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206,
242, 255, 274, 280
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-05-07 10:25:18 +02:00
|
|
|
compression_type=<str> - Compression method used for image cluster compression
|
2019-01-15 19:02:40 +01:00
|
|
|
data_file=<str> - File name of an external data file
|
2019-02-22 14:29:38 +01:00
|
|
|
data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
|
2018-10-19 18:49:25 +02:00
|
|
|
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
|
|
|
|
|
encrypt.cipher-mode=<str> - Name of encryption cipher mode
|
|
|
|
|
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
|
|
|
|
|
encrypt.hash-alg=<str> - Name of encryption hash algorithm
|
|
|
|
|
encrypt.iter-time=<num> - Time to spend in PBKDF in milliseconds
|
|
|
|
|
encrypt.ivgen-alg=<str> - Name of IV generator algorithm
|
|
|
|
|
encrypt.ivgen-hash-alg=<str> - Name of IV generator hash algorithm
|
|
|
|
|
encrypt.key-secret=<str> - ID of secret providing qcow AES key or LUKS passphrase
|
|
|
|
|
encryption=<bool (on/off)> - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes)
|
2020-07-10 18:13:13 +02:00
|
|
|
extended_l2=<bool (on/off)> - Extended L2 tables
|
file-posix: Mitigate file fragmentation with extent size hints
Especially when O_DIRECT is used with image files so that the page cache
indirection can't cause a merge of allocating requests, the file will
fragment on the file system layer, with a potentially very small
fragment size (this depends on the requests the guest sent).
On Linux, fragmentation can be reduced by setting an extent size hint
when creating the file (at least on XFS, it can't be set any more after
the first extent has been allocated), basically giving raw files a
"cluster size" for allocation.
This adds a create option to set the extent size hint, and changes the
default from not setting a hint to setting it to 1 MB. The main reason
why qcow2 defaults to smaller cluster sizes is that COW becomes more
expensive, which is not an issue with raw files, so we can choose a
larger size. The tradeoff here is only potentially wasted disk space.
For qcow2 (or other image formats) over file-posix, the advantage should
even be greater because they grow sequentially without leaving holes, so
there won't be wasted space. Setting even larger extent size hints for
such images may make sense. This can be done with the new option, but
let's keep the default conservative for now.
The effect is very visible with a test that intentionally creates a
badly fragmented file with qemu-img bench (the time difference while
creating the file is already remarkable) and then looks at the number of
extents and the time a simple "qemu-img map" takes.
Without an extent size hint:
$ ./qemu-img create -f raw -o extent_size_hint=0 ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=0
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 25.848 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 19.616 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 2000000 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m1,279s
user 0m0,043s
sys 0m1,226s
With the new default extent size hint of 1 MB:
$ ./qemu-img create -f raw -o extent_size_hint=1M ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=1048576
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 11.833 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 10.155 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 178 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m0,061s
user 0m0,040s
sys 0m0,014s
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200707142329.48303-1-kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-07 16:23:29 +02:00
|
|
|
extent_size_hint=<size> - Extent size hint for the image file, 0 to disable
|
2018-10-19 18:49:25 +02:00
|
|
|
lazy_refcounts=<bool (on/off)> - Postpone refcount updates
|
|
|
|
|
nocow=<bool (on/off)> - Turn off copy-on-write (valid only on btrfs)
|
|
|
|
|
preallocation=<str> - Preallocation mode (allowed values: off, metadata, falloc, full)
|
|
|
|
|
refcount_bits=<num> - Width of a reference count entry in bits
|
|
|
|
|
size=<size> - Virtual disk size
|
2014-02-21 16:24:08 +01:00
|
|
|
|
|
|
|
|
Testing: convert -O qcow2 -o ? TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.base
|
|
|
|
|
Supported options:
|
2018-10-19 18:49:25 +02:00
|
|
|
backing_file=<str> - File name of a base image
|
|
|
|
|
backing_fmt=<str> - Image format of the base image
|
|
|
|
|
cluster_size=<size> - qcow2 cluster size
|
2019-07-08 20:47:03 +02:00
|
|
|
compat=<str> - Compatibility level (v2 [0.10] or v3 [1.1])
|
qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206,
242, 255, 274, 280
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-05-07 10:25:18 +02:00
|
|
|
compression_type=<str> - Compression method used for image cluster compression
|
2019-01-15 19:02:40 +01:00
|
|
|
data_file=<str> - File name of an external data file
|
2019-02-22 14:29:38 +01:00
|
|
|
data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
|
2018-10-19 18:49:25 +02:00
|
|
|
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
|
|
|
|
|
encrypt.cipher-mode=<str> - Name of encryption cipher mode
|
|
|
|
|
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
|
|
|
|
|
encrypt.hash-alg=<str> - Name of encryption hash algorithm
|
|
|
|
|
encrypt.iter-time=<num> - Time to spend in PBKDF in milliseconds
|
|
|
|
|
encrypt.ivgen-alg=<str> - Name of IV generator algorithm
|
|
|
|
|
encrypt.ivgen-hash-alg=<str> - Name of IV generator hash algorithm
|
|
|
|
|
encrypt.key-secret=<str> - ID of secret providing qcow AES key or LUKS passphrase
|
|
|
|
|
encryption=<bool (on/off)> - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes)
|
2020-07-10 18:13:13 +02:00
|
|
|
extended_l2=<bool (on/off)> - Extended L2 tables
|
file-posix: Mitigate file fragmentation with extent size hints
Especially when O_DIRECT is used with image files so that the page cache
indirection can't cause a merge of allocating requests, the file will
fragment on the file system layer, with a potentially very small
fragment size (this depends on the requests the guest sent).
On Linux, fragmentation can be reduced by setting an extent size hint
when creating the file (at least on XFS, it can't be set any more after
the first extent has been allocated), basically giving raw files a
"cluster size" for allocation.
This adds a create option to set the extent size hint, and changes the
default from not setting a hint to setting it to 1 MB. The main reason
why qcow2 defaults to smaller cluster sizes is that COW becomes more
expensive, which is not an issue with raw files, so we can choose a
larger size. The tradeoff here is only potentially wasted disk space.
For qcow2 (or other image formats) over file-posix, the advantage should
even be greater because they grow sequentially without leaving holes, so
there won't be wasted space. Setting even larger extent size hints for
such images may make sense. This can be done with the new option, but
let's keep the default conservative for now.
The effect is very visible with a test that intentionally creates a
badly fragmented file with qemu-img bench (the time difference while
creating the file is already remarkable) and then looks at the number of
extents and the time a simple "qemu-img map" takes.
Without an extent size hint:
$ ./qemu-img create -f raw -o extent_size_hint=0 ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=0
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 25.848 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 19.616 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 2000000 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m1,279s
user 0m0,043s
sys 0m1,226s
With the new default extent size hint of 1 MB:
$ ./qemu-img create -f raw -o extent_size_hint=1M ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=1048576
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 11.833 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 10.155 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 178 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m0,061s
user 0m0,040s
sys 0m0,014s
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200707142329.48303-1-kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-07 16:23:29 +02:00
|
|
|
extent_size_hint=<size> - Extent size hint for the image file, 0 to disable
|
2018-10-19 18:49:25 +02:00
|
|
|
lazy_refcounts=<bool (on/off)> - Postpone refcount updates
|
|
|
|
|
nocow=<bool (on/off)> - Turn off copy-on-write (valid only on btrfs)
|
|
|
|
|
preallocation=<str> - Preallocation mode (allowed values: off, metadata, falloc, full)
|
|
|
|
|
refcount_bits=<num> - Width of a reference count entry in bits
|
|
|
|
|
size=<size> - Virtual disk size
|
2014-02-21 16:24:08 +01:00
|
|
|
|
|
|
|
|
Testing: convert -O qcow2 -o cluster_size=4k,help TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.base
|
|
|
|
|
Supported options:
|
2018-10-19 18:49:25 +02:00
|
|
|
backing_file=<str> - File name of a base image
|
|
|
|
|
backing_fmt=<str> - Image format of the base image
|
|
|
|
|
cluster_size=<size> - qcow2 cluster size
|
2019-07-08 20:47:03 +02:00
|
|
|
compat=<str> - Compatibility level (v2 [0.10] or v3 [1.1])
|
qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206,
242, 255, 274, 280
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-05-07 10:25:18 +02:00
|
|
|
compression_type=<str> - Compression method used for image cluster compression
|
2019-01-15 19:02:40 +01:00
|
|
|
data_file=<str> - File name of an external data file
|
2019-02-22 14:29:38 +01:00
|
|
|
data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
|
2018-10-19 18:49:25 +02:00
|
|
|
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
|
|
|
|
|
encrypt.cipher-mode=<str> - Name of encryption cipher mode
|
|
|
|
|
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
|
|
|
|
|
encrypt.hash-alg=<str> - Name of encryption hash algorithm
|
|
|
|
|
encrypt.iter-time=<num> - Time to spend in PBKDF in milliseconds
|
|
|
|
|
encrypt.ivgen-alg=<str> - Name of IV generator algorithm
|
|
|
|
|
encrypt.ivgen-hash-alg=<str> - Name of IV generator hash algorithm
|
|
|
|
|
encrypt.key-secret=<str> - ID of secret providing qcow AES key or LUKS passphrase
|
|
|
|
|
encryption=<bool (on/off)> - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes)
|
2020-07-10 18:13:13 +02:00
|
|
|
extended_l2=<bool (on/off)> - Extended L2 tables
|
file-posix: Mitigate file fragmentation with extent size hints
Especially when O_DIRECT is used with image files so that the page cache
indirection can't cause a merge of allocating requests, the file will
fragment on the file system layer, with a potentially very small
fragment size (this depends on the requests the guest sent).
On Linux, fragmentation can be reduced by setting an extent size hint
when creating the file (at least on XFS, it can't be set any more after
the first extent has been allocated), basically giving raw files a
"cluster size" for allocation.
This adds a create option to set the extent size hint, and changes the
default from not setting a hint to setting it to 1 MB. The main reason
why qcow2 defaults to smaller cluster sizes is that COW becomes more
expensive, which is not an issue with raw files, so we can choose a
larger size. The tradeoff here is only potentially wasted disk space.
For qcow2 (or other image formats) over file-posix, the advantage should
even be greater because they grow sequentially without leaving holes, so
there won't be wasted space. Setting even larger extent size hints for
such images may make sense. This can be done with the new option, but
let's keep the default conservative for now.
The effect is very visible with a test that intentionally creates a
badly fragmented file with qemu-img bench (the time difference while
creating the file is already remarkable) and then looks at the number of
extents and the time a simple "qemu-img map" takes.
Without an extent size hint:
$ ./qemu-img create -f raw -o extent_size_hint=0 ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=0
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 25.848 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 19.616 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 2000000 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m1,279s
user 0m0,043s
sys 0m1,226s
With the new default extent size hint of 1 MB:
$ ./qemu-img create -f raw -o extent_size_hint=1M ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=1048576
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 11.833 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 10.155 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 178 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m0,061s
user 0m0,040s
sys 0m0,014s
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200707142329.48303-1-kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-07 16:23:29 +02:00
|
|
|
extent_size_hint=<size> - Extent size hint for the image file, 0 to disable
|
2018-10-19 18:49:25 +02:00
|
|
|
lazy_refcounts=<bool (on/off)> - Postpone refcount updates
|
|
|
|
|
nocow=<bool (on/off)> - Turn off copy-on-write (valid only on btrfs)
|
|
|
|
|
preallocation=<str> - Preallocation mode (allowed values: off, metadata, falloc, full)
|
|
|
|
|
refcount_bits=<num> - Width of a reference count entry in bits
|
|
|
|
|
size=<size> - Virtual disk size
|
2014-02-21 16:24:08 +01:00
|
|
|
|
|
|
|
|
Testing: convert -O qcow2 -o cluster_size=4k,? TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.base
|
|
|
|
|
Supported options:
|
2018-10-19 18:49:25 +02:00
|
|
|
backing_file=<str> - File name of a base image
|
|
|
|
|
backing_fmt=<str> - Image format of the base image
|
|
|
|
|
cluster_size=<size> - qcow2 cluster size
|
2019-07-08 20:47:03 +02:00
|
|
|
compat=<str> - Compatibility level (v2 [0.10] or v3 [1.1])
|
qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206,
242, 255, 274, 280
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-05-07 10:25:18 +02:00
|
|
|
compression_type=<str> - Compression method used for image cluster compression
|
2019-01-15 19:02:40 +01:00
|
|
|
data_file=<str> - File name of an external data file
|
2019-02-22 14:29:38 +01:00
|
|
|
data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
|
2018-10-19 18:49:25 +02:00
|
|
|
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
|
|
|
|
|
encrypt.cipher-mode=<str> - Name of encryption cipher mode
|
|
|
|
|
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
|
|
|
|
|
encrypt.hash-alg=<str> - Name of encryption hash algorithm
|
|
|
|
|
encrypt.iter-time=<num> - Time to spend in PBKDF in milliseconds
|
|
|
|
|
encrypt.ivgen-alg=<str> - Name of IV generator algorithm
|
|
|
|
|
encrypt.ivgen-hash-alg=<str> - Name of IV generator hash algorithm
|
|
|
|
|
encrypt.key-secret=<str> - ID of secret providing qcow AES key or LUKS passphrase
|
|
|
|
|
encryption=<bool (on/off)> - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes)
|
2020-07-10 18:13:13 +02:00
|
|
|
extended_l2=<bool (on/off)> - Extended L2 tables
|
file-posix: Mitigate file fragmentation with extent size hints
Especially when O_DIRECT is used with image files so that the page cache
indirection can't cause a merge of allocating requests, the file will
fragment on the file system layer, with a potentially very small
fragment size (this depends on the requests the guest sent).
On Linux, fragmentation can be reduced by setting an extent size hint
when creating the file (at least on XFS, it can't be set any more after
the first extent has been allocated), basically giving raw files a
"cluster size" for allocation.
This adds a create option to set the extent size hint, and changes the
default from not setting a hint to setting it to 1 MB. The main reason
why qcow2 defaults to smaller cluster sizes is that COW becomes more
expensive, which is not an issue with raw files, so we can choose a
larger size. The tradeoff here is only potentially wasted disk space.
For qcow2 (or other image formats) over file-posix, the advantage should
even be greater because they grow sequentially without leaving holes, so
there won't be wasted space. Setting even larger extent size hints for
such images may make sense. This can be done with the new option, but
let's keep the default conservative for now.
The effect is very visible with a test that intentionally creates a
badly fragmented file with qemu-img bench (the time difference while
creating the file is already remarkable) and then looks at the number of
extents and the time a simple "qemu-img map" takes.
Without an extent size hint:
$ ./qemu-img create -f raw -o extent_size_hint=0 ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=0
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 25.848 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 19.616 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 2000000 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m1,279s
user 0m0,043s
sys 0m1,226s
With the new default extent size hint of 1 MB:
$ ./qemu-img create -f raw -o extent_size_hint=1M ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=1048576
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 11.833 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 10.155 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 178 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m0,061s
user 0m0,040s
sys 0m0,014s
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200707142329.48303-1-kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-07 16:23:29 +02:00
|
|
|
extent_size_hint=<size> - Extent size hint for the image file, 0 to disable
|
2018-10-19 18:49:25 +02:00
|
|
|
lazy_refcounts=<bool (on/off)> - Postpone refcount updates
|
|
|
|
|
nocow=<bool (on/off)> - Turn off copy-on-write (valid only on btrfs)
|
|
|
|
|
preallocation=<str> - Preallocation mode (allowed values: off, metadata, falloc, full)
|
|
|
|
|
refcount_bits=<num> - Width of a reference count entry in bits
|
|
|
|
|
size=<size> - Virtual disk size
|
2014-02-21 16:24:08 +01:00
|
|
|
|
|
|
|
|
Testing: convert -O qcow2 -o help,cluster_size=4k TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.base
|
|
|
|
|
Supported options:
|
2018-10-19 18:49:25 +02:00
|
|
|
backing_file=<str> - File name of a base image
|
|
|
|
|
backing_fmt=<str> - Image format of the base image
|
|
|
|
|
cluster_size=<size> - qcow2 cluster size
|
2019-07-08 20:47:03 +02:00
|
|
|
compat=<str> - Compatibility level (v2 [0.10] or v3 [1.1])
|
qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206,
242, 255, 274, 280
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-05-07 10:25:18 +02:00
|
|
|
compression_type=<str> - Compression method used for image cluster compression
|
2019-01-15 19:02:40 +01:00
|
|
|
data_file=<str> - File name of an external data file
|
2019-02-22 14:29:38 +01:00
|
|
|
data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
|
2018-10-19 18:49:25 +02:00
|
|
|
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
|
|
|
|
|
encrypt.cipher-mode=<str> - Name of encryption cipher mode
|
|
|
|
|
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
|
|
|
|
|
encrypt.hash-alg=<str> - Name of encryption hash algorithm
|
|
|
|
|
encrypt.iter-time=<num> - Time to spend in PBKDF in milliseconds
|
|
|
|
|
encrypt.ivgen-alg=<str> - Name of IV generator algorithm
|
|
|
|
|
encrypt.ivgen-hash-alg=<str> - Name of IV generator hash algorithm
|
|
|
|
|
encrypt.key-secret=<str> - ID of secret providing qcow AES key or LUKS passphrase
|
|
|
|
|
encryption=<bool (on/off)> - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes)
|
2020-07-10 18:13:13 +02:00
|
|
|
extended_l2=<bool (on/off)> - Extended L2 tables
|
file-posix: Mitigate file fragmentation with extent size hints
Especially when O_DIRECT is used with image files so that the page cache
indirection can't cause a merge of allocating requests, the file will
fragment on the file system layer, with a potentially very small
fragment size (this depends on the requests the guest sent).
On Linux, fragmentation can be reduced by setting an extent size hint
when creating the file (at least on XFS, it can't be set any more after
the first extent has been allocated), basically giving raw files a
"cluster size" for allocation.
This adds a create option to set the extent size hint, and changes the
default from not setting a hint to setting it to 1 MB. The main reason
why qcow2 defaults to smaller cluster sizes is that COW becomes more
expensive, which is not an issue with raw files, so we can choose a
larger size. The tradeoff here is only potentially wasted disk space.
For qcow2 (or other image formats) over file-posix, the advantage should
even be greater because they grow sequentially without leaving holes, so
there won't be wasted space. Setting even larger extent size hints for
such images may make sense. This can be done with the new option, but
let's keep the default conservative for now.
The effect is very visible with a test that intentionally creates a
badly fragmented file with qemu-img bench (the time difference while
creating the file is already remarkable) and then looks at the number of
extents and the time a simple "qemu-img map" takes.
Without an extent size hint:
$ ./qemu-img create -f raw -o extent_size_hint=0 ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=0
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 25.848 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 19.616 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 2000000 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m1,279s
user 0m0,043s
sys 0m1,226s
With the new default extent size hint of 1 MB:
$ ./qemu-img create -f raw -o extent_size_hint=1M ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=1048576
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 11.833 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 10.155 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 178 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m0,061s
user 0m0,040s
sys 0m0,014s
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200707142329.48303-1-kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-07 16:23:29 +02:00
|
|
|
extent_size_hint=<size> - Extent size hint for the image file, 0 to disable
|
2018-10-19 18:49:25 +02:00
|
|
|
lazy_refcounts=<bool (on/off)> - Postpone refcount updates
|
|
|
|
|
nocow=<bool (on/off)> - Turn off copy-on-write (valid only on btrfs)
|
|
|
|
|
preallocation=<str> - Preallocation mode (allowed values: off, metadata, falloc, full)
|
|
|
|
|
refcount_bits=<num> - Width of a reference count entry in bits
|
|
|
|
|
size=<size> - Virtual disk size
|
2014-02-21 16:24:08 +01:00
|
|
|
|
|
|
|
|
Testing: convert -O qcow2 -o ?,cluster_size=4k TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.base
|
|
|
|
|
Supported options:
|
2018-10-19 18:49:25 +02:00
|
|
|
backing_file=<str> - File name of a base image
|
|
|
|
|
backing_fmt=<str> - Image format of the base image
|
|
|
|
|
cluster_size=<size> - qcow2 cluster size
|
2019-07-08 20:47:03 +02:00
|
|
|
compat=<str> - Compatibility level (v2 [0.10] or v3 [1.1])
|
qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206,
242, 255, 274, 280
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-05-07 10:25:18 +02:00
|
|
|
compression_type=<str> - Compression method used for image cluster compression
|
2019-01-15 19:02:40 +01:00
|
|
|
data_file=<str> - File name of an external data file
|
2019-02-22 14:29:38 +01:00
|
|
|
data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
|
2018-10-19 18:49:25 +02:00
|
|
|
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
|
|
|
|
|
encrypt.cipher-mode=<str> - Name of encryption cipher mode
|
|
|
|
|
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
|
|
|
|
|
encrypt.hash-alg=<str> - Name of encryption hash algorithm
|
|
|
|
|
encrypt.iter-time=<num> - Time to spend in PBKDF in milliseconds
|
|
|
|
|
encrypt.ivgen-alg=<str> - Name of IV generator algorithm
|
|
|
|
|
encrypt.ivgen-hash-alg=<str> - Name of IV generator hash algorithm
|
|
|
|
|
encrypt.key-secret=<str> - ID of secret providing qcow AES key or LUKS passphrase
|
|
|
|
|
encryption=<bool (on/off)> - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes)
|
2020-07-10 18:13:13 +02:00
|
|
|
extended_l2=<bool (on/off)> - Extended L2 tables
|
file-posix: Mitigate file fragmentation with extent size hints
Especially when O_DIRECT is used with image files so that the page cache
indirection can't cause a merge of allocating requests, the file will
fragment on the file system layer, with a potentially very small
fragment size (this depends on the requests the guest sent).
On Linux, fragmentation can be reduced by setting an extent size hint
when creating the file (at least on XFS, it can't be set any more after
the first extent has been allocated), basically giving raw files a
"cluster size" for allocation.
This adds a create option to set the extent size hint, and changes the
default from not setting a hint to setting it to 1 MB. The main reason
why qcow2 defaults to smaller cluster sizes is that COW becomes more
expensive, which is not an issue with raw files, so we can choose a
larger size. The tradeoff here is only potentially wasted disk space.
For qcow2 (or other image formats) over file-posix, the advantage should
even be greater because they grow sequentially without leaving holes, so
there won't be wasted space. Setting even larger extent size hints for
such images may make sense. This can be done with the new option, but
let's keep the default conservative for now.
The effect is very visible with a test that intentionally creates a
badly fragmented file with qemu-img bench (the time difference while
creating the file is already remarkable) and then looks at the number of
extents and the time a simple "qemu-img map" takes.
Without an extent size hint:
$ ./qemu-img create -f raw -o extent_size_hint=0 ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=0
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 25.848 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 19.616 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 2000000 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m1,279s
user 0m0,043s
sys 0m1,226s
With the new default extent size hint of 1 MB:
$ ./qemu-img create -f raw -o extent_size_hint=1M ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=1048576
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 11.833 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 10.155 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 178 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m0,061s
user 0m0,040s
sys 0m0,014s
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200707142329.48303-1-kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-07 16:23:29 +02:00
|
|
|
extent_size_hint=<size> - Extent size hint for the image file, 0 to disable
|
2018-10-19 18:49:25 +02:00
|
|
|
lazy_refcounts=<bool (on/off)> - Postpone refcount updates
|
|
|
|
|
nocow=<bool (on/off)> - Turn off copy-on-write (valid only on btrfs)
|
|
|
|
|
preallocation=<str> - Preallocation mode (allowed values: off, metadata, falloc, full)
|
|
|
|
|
refcount_bits=<num> - Width of a reference count entry in bits
|
|
|
|
|
size=<size> - Virtual disk size
|
2014-02-21 16:24:08 +01:00
|
|
|
|
|
|
|
|
Testing: convert -O qcow2 -o cluster_size=4k -o help TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.base
|
|
|
|
|
Supported options:
|
2018-10-19 18:49:25 +02:00
|
|
|
backing_file=<str> - File name of a base image
|
|
|
|
|
backing_fmt=<str> - Image format of the base image
|
|
|
|
|
cluster_size=<size> - qcow2 cluster size
|
2019-07-08 20:47:03 +02:00
|
|
|
compat=<str> - Compatibility level (v2 [0.10] or v3 [1.1])
|
qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206,
242, 255, 274, 280
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-05-07 10:25:18 +02:00
|
|
|
compression_type=<str> - Compression method used for image cluster compression
|
2019-01-15 19:02:40 +01:00
|
|
|
data_file=<str> - File name of an external data file
|
2019-02-22 14:29:38 +01:00
|
|
|
data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
|
2018-10-19 18:49:25 +02:00
|
|
|
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
|
|
|
|
|
encrypt.cipher-mode=<str> - Name of encryption cipher mode
|
|
|
|
|
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
|
|
|
|
|
encrypt.hash-alg=<str> - Name of encryption hash algorithm
|
|
|
|
|
encrypt.iter-time=<num> - Time to spend in PBKDF in milliseconds
|
|
|
|
|
encrypt.ivgen-alg=<str> - Name of IV generator algorithm
|
|
|
|
|
encrypt.ivgen-hash-alg=<str> - Name of IV generator hash algorithm
|
|
|
|
|
encrypt.key-secret=<str> - ID of secret providing qcow AES key or LUKS passphrase
|
|
|
|
|
encryption=<bool (on/off)> - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes)
|
2020-07-10 18:13:13 +02:00
|
|
|
extended_l2=<bool (on/off)> - Extended L2 tables
|
file-posix: Mitigate file fragmentation with extent size hints
Especially when O_DIRECT is used with image files so that the page cache
indirection can't cause a merge of allocating requests, the file will
fragment on the file system layer, with a potentially very small
fragment size (this depends on the requests the guest sent).
On Linux, fragmentation can be reduced by setting an extent size hint
when creating the file (at least on XFS, it can't be set any more after
the first extent has been allocated), basically giving raw files a
"cluster size" for allocation.
This adds a create option to set the extent size hint, and changes the
default from not setting a hint to setting it to 1 MB. The main reason
why qcow2 defaults to smaller cluster sizes is that COW becomes more
expensive, which is not an issue with raw files, so we can choose a
larger size. The tradeoff here is only potentially wasted disk space.
For qcow2 (or other image formats) over file-posix, the advantage should
even be greater because they grow sequentially without leaving holes, so
there won't be wasted space. Setting even larger extent size hints for
such images may make sense. This can be done with the new option, but
let's keep the default conservative for now.
The effect is very visible with a test that intentionally creates a
badly fragmented file with qemu-img bench (the time difference while
creating the file is already remarkable) and then looks at the number of
extents and the time a simple "qemu-img map" takes.
Without an extent size hint:
$ ./qemu-img create -f raw -o extent_size_hint=0 ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=0
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 25.848 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 19.616 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 2000000 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m1,279s
user 0m0,043s
sys 0m1,226s
With the new default extent size hint of 1 MB:
$ ./qemu-img create -f raw -o extent_size_hint=1M ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=1048576
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 11.833 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 10.155 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 178 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m0,061s
user 0m0,040s
sys 0m0,014s
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200707142329.48303-1-kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-07 16:23:29 +02:00
|
|
|
extent_size_hint=<size> - Extent size hint for the image file, 0 to disable
|
2018-10-19 18:49:25 +02:00
|
|
|
lazy_refcounts=<bool (on/off)> - Postpone refcount updates
|
|
|
|
|
nocow=<bool (on/off)> - Turn off copy-on-write (valid only on btrfs)
|
|
|
|
|
preallocation=<str> - Preallocation mode (allowed values: off, metadata, falloc, full)
|
|
|
|
|
refcount_bits=<num> - Width of a reference count entry in bits
|
|
|
|
|
size=<size> - Virtual disk size
|
2014-02-21 16:24:08 +01:00
|
|
|
|
|
|
|
|
Testing: convert -O qcow2 -o cluster_size=4k -o ? TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.base
|
|
|
|
|
Supported options:
|
2018-10-19 18:49:25 +02:00
|
|
|
backing_file=<str> - File name of a base image
|
|
|
|
|
backing_fmt=<str> - Image format of the base image
|
|
|
|
|
cluster_size=<size> - qcow2 cluster size
|
2019-07-08 20:47:03 +02:00
|
|
|
compat=<str> - Compatibility level (v2 [0.10] or v3 [1.1])
|
qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206,
242, 255, 274, 280
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-05-07 10:25:18 +02:00
|
|
|
compression_type=<str> - Compression method used for image cluster compression
|
2019-01-15 19:02:40 +01:00
|
|
|
data_file=<str> - File name of an external data file
|
2019-02-22 14:29:38 +01:00
|
|
|
data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
|
2018-10-19 18:49:25 +02:00
|
|
|
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
|
|
|
|
|
encrypt.cipher-mode=<str> - Name of encryption cipher mode
|
|
|
|
|
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
|
|
|
|
|
encrypt.hash-alg=<str> - Name of encryption hash algorithm
|
|
|
|
|
encrypt.iter-time=<num> - Time to spend in PBKDF in milliseconds
|
|
|
|
|
encrypt.ivgen-alg=<str> - Name of IV generator algorithm
|
|
|
|
|
encrypt.ivgen-hash-alg=<str> - Name of IV generator hash algorithm
|
|
|
|
|
encrypt.key-secret=<str> - ID of secret providing qcow AES key or LUKS passphrase
|
|
|
|
|
encryption=<bool (on/off)> - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes)
|
2020-07-10 18:13:13 +02:00
|
|
|
extended_l2=<bool (on/off)> - Extended L2 tables
|
file-posix: Mitigate file fragmentation with extent size hints
Especially when O_DIRECT is used with image files so that the page cache
indirection can't cause a merge of allocating requests, the file will
fragment on the file system layer, with a potentially very small
fragment size (this depends on the requests the guest sent).
On Linux, fragmentation can be reduced by setting an extent size hint
when creating the file (at least on XFS, it can't be set any more after
the first extent has been allocated), basically giving raw files a
"cluster size" for allocation.
This adds a create option to set the extent size hint, and changes the
default from not setting a hint to setting it to 1 MB. The main reason
why qcow2 defaults to smaller cluster sizes is that COW becomes more
expensive, which is not an issue with raw files, so we can choose a
larger size. The tradeoff here is only potentially wasted disk space.
For qcow2 (or other image formats) over file-posix, the advantage should
even be greater because they grow sequentially without leaving holes, so
there won't be wasted space. Setting even larger extent size hints for
such images may make sense. This can be done with the new option, but
let's keep the default conservative for now.
The effect is very visible with a test that intentionally creates a
badly fragmented file with qemu-img bench (the time difference while
creating the file is already remarkable) and then looks at the number of
extents and the time a simple "qemu-img map" takes.
Without an extent size hint:
$ ./qemu-img create -f raw -o extent_size_hint=0 ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=0
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 25.848 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 19.616 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 2000000 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m1,279s
user 0m0,043s
sys 0m1,226s
With the new default extent size hint of 1 MB:
$ ./qemu-img create -f raw -o extent_size_hint=1M ~/tmp/test.raw 10G
Formatting '/home/kwolf/tmp/test.raw', fmt=raw size=10737418240 extent_size_hint=1048576
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 0
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 0, step size 8192)
Run completed in 11.833 seconds.
$ ./qemu-img bench -f raw -t none -n -w ~/tmp/test.raw -c 1000000 -S 8192 -o 4096
Sending 1000000 write requests, 4096 bytes each, 64 in parallel (starting at offset 4096, step size 8192)
Run completed in 10.155 seconds.
$ filefrag ~/tmp/test.raw
/home/kwolf/tmp/test.raw: 178 extents found
$ time ./qemu-img map ~/tmp/test.raw
Offset Length Mapped to File
0 0x1e8480000 0 /home/kwolf/tmp/test.raw
real 0m0,061s
user 0m0,040s
sys 0m0,014s
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Message-Id: <20200707142329.48303-1-kwolf@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-07 16:23:29 +02:00
|
|
|
extent_size_hint=<size> - Extent size hint for the image file, 0 to disable
|
2018-10-19 18:49:25 +02:00
|
|
|
lazy_refcounts=<bool (on/off)> - Postpone refcount updates
|
|
|
|
|
nocow=<bool (on/off)> - Turn off copy-on-write (valid only on btrfs)
|
|
|
|
|
preallocation=<str> - Preallocation mode (allowed values: off, metadata, falloc, full)
|
|
|
|
|
refcount_bits=<num> - Width of a reference count entry in bits
|
|
|
|
|
size=<size> - Virtual disk size
|
2014-02-21 16:24:08 +01:00
|
|
|
|
iotests: Specify explicit backing format where sensible
There are many existing qcow2 images that specify a backing file but
no format. This has been the source of CVEs in the past, but has
become more prominent of a problem now that libvirt has switched to
-blockdev. With older -drive, at least the probing was always done by
qemu (so the only risk of a changed format between successive boots of
a guest was if qemu was upgraded and probed differently). But with
newer -blockdev, libvirt must specify a format; if libvirt guesses raw
where the image was formatted, this results in data corruption visible
to the guest; conversely, if libvirt guesses qcow2 where qemu was
using raw, this can result in potential security holes, so modern
libvirt instead refuses to use images without explicit backing format.
The change in libvirt to reject images without explicit backing format
has pointed out that a number of tools have been far too reliant on
probing in the past. It's time to set a better example in our own
iotests of properly setting this parameter.
iotest calls to create, rebase, and convert are all impacted to some
degree. It's a bit annoying that we are inconsistent on command line
- while all of those accept -o backing_file=...,backing_fmt=..., the
shortcuts are different: create and rebase have -b and -F, while
convert has -B but no -F. (amend has no shortcuts, but the previous
patch just deprecated the use of amend to change backing chains).
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-9-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-06 22:39:52 +02:00
|
|
|
Testing: convert -O qcow2 -o backing_fmt=qcow2,backing_file=TEST_DIR/t.qcow2,,help TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.base
|
2014-02-21 16:24:08 +01:00
|
|
|
qemu-img: Could not open 'TEST_DIR/t.qcow2.base': Could not open backing file: Could not open 'TEST_DIR/t.qcow2,help': No such file or directory
|
|
|
|
|
|
iotests: Specify explicit backing format where sensible
There are many existing qcow2 images that specify a backing file but
no format. This has been the source of CVEs in the past, but has
become more prominent of a problem now that libvirt has switched to
-blockdev. With older -drive, at least the probing was always done by
qemu (so the only risk of a changed format between successive boots of
a guest was if qemu was upgraded and probed differently). But with
newer -blockdev, libvirt must specify a format; if libvirt guesses raw
where the image was formatted, this results in data corruption visible
to the guest; conversely, if libvirt guesses qcow2 where qemu was
using raw, this can result in potential security holes, so modern
libvirt instead refuses to use images without explicit backing format.
The change in libvirt to reject images without explicit backing format
has pointed out that a number of tools have been far too reliant on
probing in the past. It's time to set a better example in our own
iotests of properly setting this parameter.
iotest calls to create, rebase, and convert are all impacted to some
degree. It's a bit annoying that we are inconsistent on command line
- while all of those accept -o backing_file=...,backing_fmt=..., the
shortcuts are different: create and rebase have -b and -F, while
convert has -B but no -F. (amend has no shortcuts, but the previous
patch just deprecated the use of amend to change backing chains).
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-9-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-06 22:39:52 +02:00
|
|
|
Testing: convert -O qcow2 -o backing_fmt=qcow2,backing_file=TEST_DIR/t.qcow2,,? TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.base
|
2014-02-21 16:24:08 +01:00
|
|
|
qemu-img: Could not open 'TEST_DIR/t.qcow2.base': Could not open backing file: Could not open 'TEST_DIR/t.qcow2,?': No such file or directory
|
|
|
|
|
|
|
|
|
|
Testing: convert -O qcow2 -o backing_file=TEST_DIR/t.qcow2, -o help TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.base
|
|
|
|
|
qemu-img: Invalid option list: backing_file=TEST_DIR/t.qcow2,
|
|
|
|
|
|
|
|
|
|
Testing: convert -O qcow2 -o backing_file=TEST_DIR/t.qcow2 -o ,help TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.base
|
|
|
|
|
qemu-img: Invalid option list: ,help
|
|
|
|
|
|
|
|
|
|
Testing: convert -O qcow2 -o backing_file=TEST_DIR/t.qcow2 -o ,, -o help TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.base
|
|
|
|
|
qemu-img: Invalid option list: ,,
|
|
|
|
|
|
|
|
|
|
Testing: convert -O qcow2 -o help
|
2019-04-13 17:20:37 +02:00
|
|
|
Supported qcow2 options:
|
2018-10-19 18:49:25 +02:00
|
|
|
backing_file=<str> - File name of a base image
|
|
|
|
|
backing_fmt=<str> - Image format of the base image
|
|
|
|
|
cluster_size=<size> - qcow2 cluster size
|
2019-07-08 20:47:03 +02:00
|
|
|
compat=<str> - Compatibility level (v2 [0.10] or v3 [1.1])
|
qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206,
242, 255, 274, 280
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-05-07 10:25:18 +02:00
|
|
|
compression_type=<str> - Compression method used for image cluster compression
|
2019-01-15 19:02:40 +01:00
|
|
|
data_file=<str> - File name of an external data file
|
2019-02-22 14:29:38 +01:00
|
|
|
data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
|
2018-10-19 18:49:25 +02:00
|
|
|
encrypt.cipher-alg=<str> - Name of encryption cipher algorithm
|
|
|
|
|
encrypt.cipher-mode=<str> - Name of encryption cipher mode
|
|
|
|
|
encrypt.format=<str> - Encrypt the image, format choices: 'aes', 'luks'
|
|
|
|
|
encrypt.hash-alg=<str> - Name of encryption hash algorithm
|
|
|
|
|
encrypt.iter-time=<num> - Time to spend in PBKDF in milliseconds
|
|
|
|
|
encrypt.ivgen-alg=<str> - Name of IV generator algorithm
|
|
|
|
|
encrypt.ivgen-hash-alg=<str> - Name of IV generator hash algorithm
|
|
|
|
|
encrypt.key-secret=<str> - ID of secret providing qcow AES key or LUKS passphrase
|
|
|
|
|
encryption=<bool (on/off)> - Encrypt the image with format 'aes'. (Deprecated in favor of encrypt.format=aes)
|
2020-07-10 18:13:13 +02:00
|
|
|
extended_l2=<bool (on/off)> - Extended L2 tables
|
2018-10-19 18:49:25 +02:00
|
|
|
lazy_refcounts=<bool (on/off)> - Postpone refcount updates
|
|
|
|
|
preallocation=<str> - Preallocation mode (allowed values: off, metadata, falloc, full)
|
|
|
|
|
refcount_bits=<num> - Width of a reference count entry in bits
|
|
|
|
|
size=<size> - Virtual disk size
|
2014-02-21 16:24:08 +01:00
|
|
|
|
2019-04-13 17:20:37 +02:00
|
|
|
The protocol level may support further options.
|
|
|
|
|
Specify the target filename to include those options.
|
|
|
|
|
|
2014-02-21 16:24:08 +01:00
|
|
|
Testing: convert -o help
|
2019-04-13 17:20:37 +02:00
|
|
|
Supported raw options:
|
2018-10-19 18:49:25 +02:00
|
|
|
size=<size> - Virtual disk size
|
2014-02-21 16:24:08 +01:00
|
|
|
|
2019-04-13 17:20:37 +02:00
|
|
|
The protocol level may support further options.
|
|
|
|
|
Specify the target filename to include those options.
|
|
|
|
|
|
2018-05-09 23:00:22 +02:00
|
|
|
Testing: convert -O bochs -o help
|
|
|
|
|
qemu-img: Format driver 'bochs' does not support image creation
|
|
|
|
|
|
2018-07-27 05:34:02 +02:00
|
|
|
=== convert: -C and other options ===
|
|
|
|
|
|
|
|
|
|
Testing: convert -C -S 4k -O qcow2 TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.target
|
|
|
|
|
qemu-img: Cannot enable copy offloading when -S is used
|
|
|
|
|
|
|
|
|
|
Testing: convert -C -S 8k -O qcow2 TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.target
|
|
|
|
|
qemu-img: Cannot enable copy offloading when -S is used
|
|
|
|
|
|
|
|
|
|
Testing: convert -C -c -O qcow2 TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.target
|
|
|
|
|
qemu-img: Cannot enable copy offloading when -c is used
|
|
|
|
|
|
2019-05-07 22:35:08 +02:00
|
|
|
Testing: convert -C --salvage -O qcow2 TEST_DIR/t.qcow2 TEST_DIR/t.qcow2.target
|
|
|
|
|
qemu-img: Cannot use copy offloading in salvaging mode
|
|
|
|
|
|
2014-02-21 16:24:08 +01:00
|
|
|
=== amend: Options specified more than once ===
|
|
|
|
|
|
|
|
|
|
Testing: amend -f foo -f qcow2 -o lazy_refcounts=on TEST_DIR/t.qcow2
|
2014-11-27 17:28:01 +01:00
|
|
|
image: TEST_DIR/t.IMGFMT
|
|
|
|
|
file format: IMGFMT
|
2019-04-17 19:11:01 +02:00
|
|
|
virtual size: 128 MiB (134217728 bytes)
|
2014-02-21 16:24:08 +01:00
|
|
|
cluster_size: 65536
|
|
|
|
|
Format specific information:
|
|
|
|
|
compat: 1.1
|
qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206,
242, 255, 274, 280
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-05-07 10:25:18 +02:00
|
|
|
compression type: zlib
|
2014-02-21 16:24:08 +01:00
|
|
|
lazy refcounts: true
|
2015-02-10 21:28:44 +01:00
|
|
|
refcount bits: 16
|
2014-09-30 21:31:28 +02:00
|
|
|
corrupt: false
|
2020-07-10 18:13:13 +02:00
|
|
|
extended l2: false
|
2014-02-21 16:24:08 +01:00
|
|
|
|
|
|
|
|
Testing: amend -f qcow2 -o size=130M -o lazy_refcounts=off TEST_DIR/t.qcow2
|
2014-11-27 17:28:01 +01:00
|
|
|
image: TEST_DIR/t.IMGFMT
|
|
|
|
|
file format: IMGFMT
|
2019-04-17 19:11:01 +02:00
|
|
|
virtual size: 130 MiB (136314880 bytes)
|
2014-02-21 16:24:08 +01:00
|
|
|
cluster_size: 65536
|
|
|
|
|
Format specific information:
|
|
|
|
|
compat: 1.1
|
qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206,
242, 255, 274, 280
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-05-07 10:25:18 +02:00
|
|
|
compression type: zlib
|
2014-02-21 16:24:08 +01:00
|
|
|
lazy refcounts: false
|
2015-02-10 21:28:44 +01:00
|
|
|
refcount bits: 16
|
2014-09-30 21:31:28 +02:00
|
|
|
corrupt: false
|
2020-07-10 18:13:13 +02:00
|
|
|
extended l2: false
|
2014-02-21 16:24:08 +01:00
|
|
|
|
|
|
|
|
Testing: amend -f qcow2 -o size=8M -o lazy_refcounts=on -o size=132M TEST_DIR/t.qcow2
|
2014-11-27 17:28:01 +01:00
|
|
|
image: TEST_DIR/t.IMGFMT
|
|
|
|
|
file format: IMGFMT
|
2019-04-17 19:11:01 +02:00
|
|
|
virtual size: 132 MiB (138412032 bytes)
|
2014-02-21 16:24:08 +01:00
|
|
|
cluster_size: 65536
|
|
|
|
|
Format specific information:
|
|
|
|
|
compat: 1.1
|
qcow2: introduce compression type feature
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 082, 085, 144, 182, 185, 198, 206,
242, 255, 274, 280
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
Message-Id: <20200507082521.29210-2-dplotnikov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2020-05-07 10:25:18 +02:00
|
|
|
compression type: zlib
|
2014-02-21 16:24:08 +01:00
|
|
|
lazy refcounts: true
|
2015-02-10 21:28:44 +01:00
|
|
|
refcount bits: 16
|
2014-09-30 21:31:28 +02:00
|
|
|
corrupt: false
|
2020-07-10 18:13:13 +02:00
|
|
|
extended l2: false
|
2014-02-21 16:24:08 +01:00
|
|
|
|
|
|
|
|
Testing: amend -f qcow2 -o size=4M,size=148M TEST_DIR/t.qcow2
|
2014-09-30 21:31:27 +02:00
|
|
|
image: TEST_DIR/t.IMGFMT
|
|
|
|
|
file format: IMGFMT
|
2019-04-17 19:11:01 +02:00
|
|
|
virtual size: 148 MiB (155189248 bytes)
|
2014-02-21 16:24:08 +01:00
|
|
|
cluster_size: 65536
|
|
|
|
|
|
|
|
|
|
=== amend: help for -o ===
|
|
|
|
|
|
|
|
|
|
Testing: amend -f qcow2 -o help TEST_DIR/t.qcow2
|
2020-06-25 14:55:40 +02:00
|
|
|
Amend options for 'qcow2':
|
2018-10-19 18:49:25 +02:00
|
|
|
backing_file=<str> - File name of a base image
|
|
|
|
|
backing_fmt=<str> - Image format of the base image
|
2019-07-08 20:47:03 +02:00
|
|
|
compat=<str> - Compatibility level (v2 [0.10] or v3 [1.1])
|
2019-01-15 19:02:40 +01:00
|
|
|
data_file=<str> - File name of an external data file
|
2019-02-22 14:29:38 +01:00
|
|
|
data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
|
2020-06-25 14:55:43 +02:00
|
|
|
encrypt.iter-time=<num> - Time to spend in PBKDF in milliseconds
|
|
|
|
|
encrypt.keyslot=<num> - Select a single keyslot to modify explicitly
|
|
|
|
|
encrypt.new-secret=<str> - New secret to set in the matching keyslots. Empty string to erase
|
|
|
|
|
encrypt.old-secret=<str> - Select all keyslots that match this password
|
|
|
|
|
encrypt.state=<str> - Select new state of affected keyslots (active/inactive)
|
2018-10-19 18:49:25 +02:00
|
|
|
lazy_refcounts=<bool (on/off)> - Postpone refcount updates
|
|
|
|
|
refcount_bits=<num> - Width of a reference count entry in bits
|
|
|
|
|
size=<size> - Virtual disk size
|
2018-05-09 23:00:20 +02:00
|
|
|
|
2014-02-21 16:24:08 +01:00
|
|
|
Testing: amend -f qcow2 -o ? TEST_DIR/t.qcow2
|
2020-06-25 14:55:40 +02:00
|
|
|
Amend options for 'qcow2':
|
2018-10-19 18:49:25 +02:00
|
|
|
backing_file=<str> - File name of a base image
|
|
|
|
|
backing_fmt=<str> - Image format of the base image
|
2019-07-08 20:47:03 +02:00
|
|
|
compat=<str> - Compatibility level (v2 [0.10] or v3 [1.1])
|
2019-01-15 19:02:40 +01:00
|
|
|
data_file=<str> - File name of an external data file
|
2019-02-22 14:29:38 +01:00
|
|
|
data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
|
2020-06-25 14:55:43 +02:00
|
|
|
encrypt.iter-time=<num> - Time to spend in PBKDF in milliseconds
|
|
|
|
|
encrypt.keyslot=<num> - Select a single keyslot to modify explicitly
|
|
|
|
|
encrypt.new-secret=<str> - New secret to set in the matching keyslots. Empty string to erase
|
|
|
|
|
encrypt.old-secret=<str> - Select all keyslots that match this password
|
|
|
|
|
encrypt.state=<str> - Select new state of affected keyslots (active/inactive)
|
2018-10-19 18:49:25 +02:00
|
|
|
lazy_refcounts=<bool (on/off)> - Postpone refcount updates
|
|
|
|
|
refcount_bits=<num> - Width of a reference count entry in bits
|
|
|
|
|
size=<size> - Virtual disk size
|
2018-05-09 23:00:20 +02:00
|
|
|
|
2014-02-21 16:24:08 +01:00
|
|
|
Testing: amend -f qcow2 -o cluster_size=4k,help TEST_DIR/t.qcow2
|
2020-06-25 14:55:40 +02:00
|
|
|
Amend options for 'qcow2':
|
2018-10-19 18:49:25 +02:00
|
|
|
backing_file=<str> - File name of a base image
|
|
|
|
|
backing_fmt=<str> - Image format of the base image
|
2019-07-08 20:47:03 +02:00
|
|
|
compat=<str> - Compatibility level (v2 [0.10] or v3 [1.1])
|
2019-01-15 19:02:40 +01:00
|
|
|
data_file=<str> - File name of an external data file
|
2019-02-22 14:29:38 +01:00
|
|
|
data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
|
2020-06-25 14:55:43 +02:00
|
|
|
encrypt.iter-time=<num> - Time to spend in PBKDF in milliseconds
|
|
|
|
|
encrypt.keyslot=<num> - Select a single keyslot to modify explicitly
|
|
|
|
|
encrypt.new-secret=<str> - New secret to set in the matching keyslots. Empty string to erase
|
|
|
|
|
encrypt.old-secret=<str> - Select all keyslots that match this password
|
|
|
|
|
encrypt.state=<str> - Select new state of affected keyslots (active/inactive)
|
2018-10-19 18:49:25 +02:00
|
|
|
lazy_refcounts=<bool (on/off)> - Postpone refcount updates
|
|
|
|
|
refcount_bits=<num> - Width of a reference count entry in bits
|
|
|
|
|
size=<size> - Virtual disk size
|
2018-05-09 23:00:20 +02:00
|
|
|
|
2014-02-21 16:24:08 +01:00
|
|
|
Testing: amend -f qcow2 -o cluster_size=4k,? TEST_DIR/t.qcow2
|
2020-06-25 14:55:40 +02:00
|
|
|
Amend options for 'qcow2':
|
2018-10-19 18:49:25 +02:00
|
|
|
backing_file=<str> - File name of a base image
|
|
|
|
|
backing_fmt=<str> - Image format of the base image
|
2019-07-08 20:47:03 +02:00
|
|
|
compat=<str> - Compatibility level (v2 [0.10] or v3 [1.1])
|
2019-01-15 19:02:40 +01:00
|
|
|
data_file=<str> - File name of an external data file
|
2019-02-22 14:29:38 +01:00
|
|
|
data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
|
2020-06-25 14:55:43 +02:00
|
|
|
encrypt.iter-time=<num> - Time to spend in PBKDF in milliseconds
|
|
|
|
|
encrypt.keyslot=<num> - Select a single keyslot to modify explicitly
|
|
|
|
|
encrypt.new-secret=<str> - New secret to set in the matching keyslots. Empty string to erase
|
|
|
|
|
encrypt.old-secret=<str> - Select all keyslots that match this password
|
|
|
|
|
encrypt.state=<str> - Select new state of affected keyslots (active/inactive)
|
2018-10-19 18:49:25 +02:00
|
|
|
lazy_refcounts=<bool (on/off)> - Postpone refcount updates
|
|
|
|
|
refcount_bits=<num> - Width of a reference count entry in bits
|
|
|
|
|
size=<size> - Virtual disk size
|
2018-05-09 23:00:20 +02:00
|
|
|
|
2014-02-21 16:24:08 +01:00
|
|
|
Testing: amend -f qcow2 -o help,cluster_size=4k TEST_DIR/t.qcow2
|
2020-06-25 14:55:40 +02:00
|
|
|
Amend options for 'qcow2':
|
2018-10-19 18:49:25 +02:00
|
|
|
backing_file=<str> - File name of a base image
|
|
|
|
|
backing_fmt=<str> - Image format of the base image
|
2019-07-08 20:47:03 +02:00
|
|
|
compat=<str> - Compatibility level (v2 [0.10] or v3 [1.1])
|
2019-01-15 19:02:40 +01:00
|
|
|
data_file=<str> - File name of an external data file
|
2019-02-22 14:29:38 +01:00
|
|
|
data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
|
2020-06-25 14:55:43 +02:00
|
|
|
encrypt.iter-time=<num> - Time to spend in PBKDF in milliseconds
|
|
|
|
|
encrypt.keyslot=<num> - Select a single keyslot to modify explicitly
|
|
|
|
|
encrypt.new-secret=<str> - New secret to set in the matching keyslots. Empty string to erase
|
|
|
|
|
encrypt.old-secret=<str> - Select all keyslots that match this password
|
|
|
|
|
encrypt.state=<str> - Select new state of affected keyslots (active/inactive)
|
2018-10-19 18:49:25 +02:00
|
|
|
lazy_refcounts=<bool (on/off)> - Postpone refcount updates
|
|
|
|
|
refcount_bits=<num> - Width of a reference count entry in bits
|
|
|
|
|
size=<size> - Virtual disk size
|
2018-05-09 23:00:20 +02:00
|
|
|
|
2014-02-21 16:24:08 +01:00
|
|
|
Testing: amend -f qcow2 -o ?,cluster_size=4k TEST_DIR/t.qcow2
|
2020-06-25 14:55:40 +02:00
|
|
|
Amend options for 'qcow2':
|
2018-10-19 18:49:25 +02:00
|
|
|
backing_file=<str> - File name of a base image
|
|
|
|
|
backing_fmt=<str> - Image format of the base image
|
2019-07-08 20:47:03 +02:00
|
|
|
compat=<str> - Compatibility level (v2 [0.10] or v3 [1.1])
|
2019-01-15 19:02:40 +01:00
|
|
|
data_file=<str> - File name of an external data file
|
2019-02-22 14:29:38 +01:00
|
|
|
data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
|
2020-06-25 14:55:43 +02:00
|
|
|
encrypt.iter-time=<num> - Time to spend in PBKDF in milliseconds
|
|
|
|
|
encrypt.keyslot=<num> - Select a single keyslot to modify explicitly
|
|
|
|
|
encrypt.new-secret=<str> - New secret to set in the matching keyslots. Empty string to erase
|
|
|
|
|
encrypt.old-secret=<str> - Select all keyslots that match this password
|
|
|
|
|
encrypt.state=<str> - Select new state of affected keyslots (active/inactive)
|
2018-10-19 18:49:25 +02:00
|
|
|
lazy_refcounts=<bool (on/off)> - Postpone refcount updates
|
|
|
|
|
refcount_bits=<num> - Width of a reference count entry in bits
|
|
|
|
|
size=<size> - Virtual disk size
|
2018-05-09 23:00:20 +02:00
|
|
|
|
2014-02-21 16:24:08 +01:00
|
|
|
Testing: amend -f qcow2 -o cluster_size=4k -o help TEST_DIR/t.qcow2
|
2020-06-25 14:55:40 +02:00
|
|
|
Amend options for 'qcow2':
|
2018-10-19 18:49:25 +02:00
|
|
|
backing_file=<str> - File name of a base image
|
|
|
|
|
backing_fmt=<str> - Image format of the base image
|
2019-07-08 20:47:03 +02:00
|
|
|
compat=<str> - Compatibility level (v2 [0.10] or v3 [1.1])
|
2019-01-15 19:02:40 +01:00
|
|
|
data_file=<str> - File name of an external data file
|
2019-02-22 14:29:38 +01:00
|
|
|
data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
|
2020-06-25 14:55:43 +02:00
|
|
|
encrypt.iter-time=<num> - Time to spend in PBKDF in milliseconds
|
|
|
|
|
encrypt.keyslot=<num> - Select a single keyslot to modify explicitly
|
|
|
|
|
encrypt.new-secret=<str> - New secret to set in the matching keyslots. Empty string to erase
|
|
|
|
|
encrypt.old-secret=<str> - Select all keyslots that match this password
|
|
|
|
|
encrypt.state=<str> - Select new state of affected keyslots (active/inactive)
|
2018-10-19 18:49:25 +02:00
|
|
|
lazy_refcounts=<bool (on/off)> - Postpone refcount updates
|
|
|
|
|
refcount_bits=<num> - Width of a reference count entry in bits
|
|
|
|
|
size=<size> - Virtual disk size
|
2018-05-09 23:00:20 +02:00
|
|
|
|
2014-02-21 16:24:08 +01:00
|
|
|
Testing: amend -f qcow2 -o cluster_size=4k -o ? TEST_DIR/t.qcow2
|
2020-06-25 14:55:40 +02:00
|
|
|
Amend options for 'qcow2':
|
2018-10-19 18:49:25 +02:00
|
|
|
backing_file=<str> - File name of a base image
|
|
|
|
|
backing_fmt=<str> - Image format of the base image
|
2019-07-08 20:47:03 +02:00
|
|
|
compat=<str> - Compatibility level (v2 [0.10] or v3 [1.1])
|
2019-01-15 19:02:40 +01:00
|
|
|
data_file=<str> - File name of an external data file
|
2019-02-22 14:29:38 +01:00
|
|
|
data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
|
2020-06-25 14:55:43 +02:00
|
|
|
encrypt.iter-time=<num> - Time to spend in PBKDF in milliseconds
|
|
|
|
|
encrypt.keyslot=<num> - Select a single keyslot to modify explicitly
|
|
|
|
|
encrypt.new-secret=<str> - New secret to set in the matching keyslots. Empty string to erase
|
|
|
|
|
encrypt.old-secret=<str> - Select all keyslots that match this password
|
|
|
|
|
encrypt.state=<str> - Select new state of affected keyslots (active/inactive)
|
2018-10-19 18:49:25 +02:00
|
|
|
lazy_refcounts=<bool (on/off)> - Postpone refcount updates
|
|
|
|
|
refcount_bits=<num> - Width of a reference count entry in bits
|
|
|
|
|
size=<size> - Virtual disk size
|
2018-05-09 23:00:20 +02:00
|
|
|
|
2014-02-21 16:24:08 +01:00
|
|
|
Testing: amend -f qcow2 -o backing_file=TEST_DIR/t.qcow2,,help TEST_DIR/t.qcow2
|
qcow2: Deprecate use of qemu-img amend to change backing file
The use of 'qemu-img amend' to change qcow2 backing files is not
tested very well. In particular, our implementation has a bug where
if a new backing file is provided without a format, then the prior
format is blindly reused, even if this results in data corruption, but
this is not caught by iotests.
There are also situations where amending other options needs access to
the original backing file (for example, on a downgrade to a v2 image,
knowing whether a v3 zero cluster must be allocated or may be left
unallocated depends on knowing whether the backing file already reads
as zero), but the command line does not have a nice way to tell us
both the backing file to use for opening the image as well as the
backing file to install after the operation is complete.
Even if we do allow changing the backing file, it is redundant with
the existing ability to change backing files via 'qemu-img rebase -u'.
It is time to deprecate this support (leaving the existing behavior
intact, even if it is buggy), and at a point in the future, require
the use of only 'qemu-img rebase' for adjusting backing chain
relations, saving 'qemu-img amend' for changes unrelated to the
backing chain.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-8-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-06 22:39:51 +02:00
|
|
|
qemu-img: warning: Deprecated use of amend to alter the backing file; use qemu-img rebase instead
|
2014-02-21 16:24:08 +01:00
|
|
|
|
|
|
|
|
Testing: rebase -u -b -f qcow2 TEST_DIR/t.qcow2
|
|
|
|
|
|
|
|
|
|
Testing: amend -f qcow2 -o backing_file=TEST_DIR/t.qcow2,,? TEST_DIR/t.qcow2
|
qcow2: Deprecate use of qemu-img amend to change backing file
The use of 'qemu-img amend' to change qcow2 backing files is not
tested very well. In particular, our implementation has a bug where
if a new backing file is provided without a format, then the prior
format is blindly reused, even if this results in data corruption, but
this is not caught by iotests.
There are also situations where amending other options needs access to
the original backing file (for example, on a downgrade to a v2 image,
knowing whether a v3 zero cluster must be allocated or may be left
unallocated depends on knowing whether the backing file already reads
as zero), but the command line does not have a nice way to tell us
both the backing file to use for opening the image as well as the
backing file to install after the operation is complete.
Even if we do allow changing the backing file, it is redundant with
the existing ability to change backing files via 'qemu-img rebase -u'.
It is time to deprecate this support (leaving the existing behavior
intact, even if it is buggy), and at a point in the future, require
the use of only 'qemu-img rebase' for adjusting backing chain
relations, saving 'qemu-img amend' for changes unrelated to the
backing chain.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200706203954.341758-8-eblake@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2020-07-06 22:39:51 +02:00
|
|
|
qemu-img: warning: Deprecated use of amend to alter the backing file; use qemu-img rebase instead
|
2014-02-21 16:24:08 +01:00
|
|
|
|
|
|
|
|
Testing: rebase -u -b -f qcow2 TEST_DIR/t.qcow2
|
|
|
|
|
|
|
|
|
|
Testing: amend -f qcow2 -o backing_file=TEST_DIR/t.qcow2, -o help TEST_DIR/t.qcow2
|
|
|
|
|
qemu-img: Invalid option list: backing_file=TEST_DIR/t.qcow2,
|
|
|
|
|
|
|
|
|
|
Testing: amend -f qcow2 -o backing_file=TEST_DIR/t.qcow2 -o ,help TEST_DIR/t.qcow2
|
|
|
|
|
qemu-img: Invalid option list: ,help
|
|
|
|
|
|
|
|
|
|
Testing: amend -f qcow2 -o backing_file=TEST_DIR/t.qcow2 -o ,, -o help TEST_DIR/t.qcow2
|
|
|
|
|
qemu-img: Invalid option list: ,,
|
|
|
|
|
|
|
|
|
|
Testing: amend -f qcow2 -o help
|
2020-06-25 14:55:40 +02:00
|
|
|
Amend options for 'qcow2':
|
2018-10-19 18:49:25 +02:00
|
|
|
backing_file=<str> - File name of a base image
|
|
|
|
|
backing_fmt=<str> - Image format of the base image
|
2019-07-08 20:47:03 +02:00
|
|
|
compat=<str> - Compatibility level (v2 [0.10] or v3 [1.1])
|
2019-01-15 19:02:40 +01:00
|
|
|
data_file=<str> - File name of an external data file
|
2019-02-22 14:29:38 +01:00
|
|
|
data_file_raw=<bool (on/off)> - The external data file must stay valid as a raw image
|
2020-06-25 14:55:43 +02:00
|
|
|
encrypt.iter-time=<num> - Time to spend in PBKDF in milliseconds
|
|
|
|
|
encrypt.keyslot=<num> - Select a single keyslot to modify explicitly
|
|
|
|
|
encrypt.new-secret=<str> - New secret to set in the matching keyslots. Empty string to erase
|
|
|
|
|
encrypt.old-secret=<str> - Select all keyslots that match this password
|
|
|
|
|
encrypt.state=<str> - Select new state of affected keyslots (active/inactive)
|
2018-10-19 18:49:25 +02:00
|
|
|
lazy_refcounts=<bool (on/off)> - Postpone refcount updates
|
|
|
|
|
refcount_bits=<num> - Width of a reference count entry in bits
|
|
|
|
|
size=<size> - Virtual disk size
|
2014-02-21 16:24:08 +01:00
|
|
|
|
2019-04-13 17:20:36 +02:00
|
|
|
Testing: amend -o help
|
|
|
|
|
qemu-img: Expecting one image file name
|
2018-05-09 23:00:22 +02:00
|
|
|
|
|
|
|
|
Testing: amend -f bochs -o help
|
|
|
|
|
qemu-img: Format driver 'bochs' does not support option amendment
|
2014-02-21 16:24:08 +01:00
|
|
|
*** done
|
