87bf54bb43
Add zstd compression and decompression support to SquashFS. zstd is a great fit for SquashFS because it can compress at ratios approaching xz, while decompressing twice as fast as zlib. For SquashFS in particular, it can decompress as fast as lzo and lz4. It also has the flexibility to turn down the compression ratio for faster compression times. The compression benchmark is run on the file tree from the SquashFS archive found in ubuntu-16.10-desktop-amd64.iso [1]. It uses `mksquashfs` with the default block size (128 KB) and and various compression algorithms/levels. xz and zstd are also benchmarked with 256 KB blocks. The decompression benchmark times how long it takes to `tar` the file tree into `/dev/null`. See the benchmark file in the upstream zstd source repository located under `contrib/linux-kernel/squashfs-benchmark.sh` [2] for details. I ran the benchmarks on a Ubuntu 14.04 VM with 2 cores and 4 GiB of RAM. The VM is running on a MacBook Pro with a 3.1 GHz Intel Core i7 processor, 16 GB of RAM, and a SSD. | Method | Ratio | Compression MB/s | Decompression MB/s | |----------------|-------|------------------|--------------------| | gzip | 2.92 | 15 | 128 | | lzo | 2.64 | 9.5 | 217 | | lz4 | 2.12 | 94 | 218 | | xz | 3.43 | 5.5 | 35 | | xz 256 KB | 3.53 | 5.4 | 40 | | zstd 1 | 2.71 | 96 | 210 | | zstd 5 | 2.93 | 69 | 198 | | zstd 10 | 3.01 | 41 | 225 | | zstd 15 | 3.13 | 11.4 | 224 | | zstd 16 256 KB | 3.24 | 8.1 | 210 | This patch was written by Sean Purcell <me@seanp.xyz>, but I will be taking over the submission process. [1] http://releases.ubuntu.com/16.10/ [2] https://github.com/facebook/zstd/blob/dev/contrib/linux-kernel/squashfs-benchmark.sh zstd source repository: https://github.com/facebook/zstd Signed-off-by: Sean Purcell <me@seanp.xyz> Signed-off-by: Nick Terrell <terrelln@fb.com> Signed-off-by: Chris Mason <clm@fb.com> Acked-by: Phillip Lougher <phillip@squashfs.org.uk>
225 lines
7.8 KiB
Plaintext
225 lines
7.8 KiB
Plaintext
config SQUASHFS
|
|
tristate "SquashFS 4.0 - Squashed file system support"
|
|
depends on BLOCK
|
|
help
|
|
Saying Y here includes support for SquashFS 4.0 (a Compressed
|
|
Read-Only File System). Squashfs is a highly compressed read-only
|
|
filesystem for Linux. It uses zlib, lzo or xz compression to
|
|
compress both files, inodes and directories. Inodes in the system
|
|
are very small and all blocks are packed to minimise data overhead.
|
|
Block sizes greater than 4K are supported up to a maximum of 1 Mbytes
|
|
(default block size 128K). SquashFS 4.0 supports 64 bit filesystems
|
|
and files (larger than 4GB), full uid/gid information, hard links and
|
|
timestamps.
|
|
|
|
Squashfs is intended for general read-only filesystem use, for
|
|
archival use (i.e. in cases where a .tar.gz file may be used), and in
|
|
embedded systems where low overhead is needed. Further information
|
|
and tools are available from http://squashfs.sourceforge.net.
|
|
|
|
If you want to compile this as a module ( = code which can be
|
|
inserted in and removed from the running kernel whenever you want),
|
|
say M here. The module will be called squashfs. Note that the root
|
|
file system (the one containing the directory /) cannot be compiled
|
|
as a module.
|
|
|
|
If unsure, say N.
|
|
|
|
choice
|
|
prompt "File decompression options"
|
|
depends on SQUASHFS
|
|
help
|
|
Squashfs now supports two options for decompressing file
|
|
data. Traditionally Squashfs has decompressed into an
|
|
intermediate buffer and then memcopied it into the page cache.
|
|
Squashfs now supports the ability to decompress directly into
|
|
the page cache.
|
|
|
|
If unsure, select "Decompress file data into an intermediate buffer"
|
|
|
|
config SQUASHFS_FILE_CACHE
|
|
bool "Decompress file data into an intermediate buffer"
|
|
help
|
|
Decompress file data into an intermediate buffer and then
|
|
memcopy it into the page cache.
|
|
|
|
config SQUASHFS_FILE_DIRECT
|
|
bool "Decompress files directly into the page cache"
|
|
help
|
|
Directly decompress file data into the page cache.
|
|
Doing so can significantly improve performance because
|
|
it eliminates a memcpy and it also removes the lock contention
|
|
on the single buffer.
|
|
|
|
endchoice
|
|
|
|
choice
|
|
prompt "Decompressor parallelisation options"
|
|
depends on SQUASHFS
|
|
help
|
|
Squashfs now supports three parallelisation options for
|
|
decompression. Each one exhibits various trade-offs between
|
|
decompression performance and CPU and memory usage.
|
|
|
|
If in doubt, select "Single threaded compression"
|
|
|
|
config SQUASHFS_DECOMP_SINGLE
|
|
bool "Single threaded compression"
|
|
help
|
|
Traditionally Squashfs has used single-threaded decompression.
|
|
Only one block (data or metadata) can be decompressed at any
|
|
one time. This limits CPU and memory usage to a minimum.
|
|
|
|
config SQUASHFS_DECOMP_MULTI
|
|
bool "Use multiple decompressors for parallel I/O"
|
|
help
|
|
By default Squashfs uses a single decompressor but it gives
|
|
poor performance on parallel I/O workloads when using multiple CPU
|
|
machines due to waiting on decompressor availability.
|
|
|
|
If you have a parallel I/O workload and your system has enough memory,
|
|
using this option may improve overall I/O performance.
|
|
|
|
This decompressor implementation uses up to two parallel
|
|
decompressors per core. It dynamically allocates decompressors
|
|
on a demand basis.
|
|
|
|
config SQUASHFS_DECOMP_MULTI_PERCPU
|
|
bool "Use percpu multiple decompressors for parallel I/O"
|
|
help
|
|
By default Squashfs uses a single decompressor but it gives
|
|
poor performance on parallel I/O workloads when using multiple CPU
|
|
machines due to waiting on decompressor availability.
|
|
|
|
This decompressor implementation uses a maximum of one
|
|
decompressor per core. It uses percpu variables to ensure
|
|
decompression is load-balanced across the cores.
|
|
|
|
endchoice
|
|
|
|
config SQUASHFS_XATTR
|
|
bool "Squashfs XATTR support"
|
|
depends on SQUASHFS
|
|
help
|
|
Saying Y here includes support for extended attributes (xattrs).
|
|
Xattrs are name:value pairs associated with inodes by
|
|
the kernel or by users (see the attr(5) manual page).
|
|
|
|
If unsure, say N.
|
|
|
|
config SQUASHFS_ZLIB
|
|
bool "Include support for ZLIB compressed file systems"
|
|
depends on SQUASHFS
|
|
select ZLIB_INFLATE
|
|
default y
|
|
help
|
|
ZLIB compression is the standard compression used by Squashfs
|
|
file systems. It offers a good trade-off between compression
|
|
achieved and the amount of CPU time and memory necessary to
|
|
compress and decompress.
|
|
|
|
If unsure, say Y.
|
|
|
|
config SQUASHFS_LZ4
|
|
bool "Include support for LZ4 compressed file systems"
|
|
depends on SQUASHFS
|
|
select LZ4_DECOMPRESS
|
|
help
|
|
Saying Y here includes support for reading Squashfs file systems
|
|
compressed with LZ4 compression. LZ4 compression is mainly
|
|
aimed at embedded systems with slower CPUs where the overheads
|
|
of zlib are too high.
|
|
|
|
LZ4 is not the standard compression used in Squashfs and so most
|
|
file systems will be readable without selecting this option.
|
|
|
|
If unsure, say N.
|
|
|
|
config SQUASHFS_LZO
|
|
bool "Include support for LZO compressed file systems"
|
|
depends on SQUASHFS
|
|
select LZO_DECOMPRESS
|
|
help
|
|
Saying Y here includes support for reading Squashfs file systems
|
|
compressed with LZO compression. LZO compression is mainly
|
|
aimed at embedded systems with slower CPUs where the overheads
|
|
of zlib are too high.
|
|
|
|
LZO is not the standard compression used in Squashfs and so most
|
|
file systems will be readable without selecting this option.
|
|
|
|
If unsure, say N.
|
|
|
|
config SQUASHFS_XZ
|
|
bool "Include support for XZ compressed file systems"
|
|
depends on SQUASHFS
|
|
select XZ_DEC
|
|
help
|
|
Saying Y here includes support for reading Squashfs file systems
|
|
compressed with XZ compression. XZ gives better compression than
|
|
the default zlib compression, at the expense of greater CPU and
|
|
memory overhead.
|
|
|
|
XZ is not the standard compression used in Squashfs and so most
|
|
file systems will be readable without selecting this option.
|
|
|
|
If unsure, say N.
|
|
|
|
config SQUASHFS_ZSTD
|
|
bool "Include support for ZSTD compressed file systems"
|
|
depends on SQUASHFS
|
|
select ZSTD_DECOMPRESS
|
|
help
|
|
Saying Y here includes support for reading Squashfs file systems
|
|
compressed with ZSTD compression. ZSTD gives better compression than
|
|
the default ZLIB compression, while using less CPU.
|
|
|
|
ZSTD is not the standard compression used in Squashfs and so most
|
|
file systems will be readable without selecting this option.
|
|
|
|
If unsure, say N.
|
|
|
|
config SQUASHFS_4K_DEVBLK_SIZE
|
|
bool "Use 4K device block size?"
|
|
depends on SQUASHFS
|
|
help
|
|
By default Squashfs sets the dev block size (sb_min_blocksize)
|
|
to 1K or the smallest block size supported by the block device
|
|
(if larger). This, because blocks are packed together and
|
|
unaligned in Squashfs, should reduce latency.
|
|
|
|
This, however, gives poor performance on MTD NAND devices where
|
|
the optimal I/O size is 4K (even though the devices can support
|
|
smaller block sizes).
|
|
|
|
Using a 4K device block size may also improve overall I/O
|
|
performance for some file access patterns (e.g. sequential
|
|
accesses of files in filesystem order) on all media.
|
|
|
|
Setting this option will force Squashfs to use a 4K device block
|
|
size by default.
|
|
|
|
If unsure, say N.
|
|
|
|
config SQUASHFS_EMBEDDED
|
|
bool "Additional option for memory-constrained systems"
|
|
depends on SQUASHFS
|
|
help
|
|
Saying Y here allows you to specify cache size.
|
|
|
|
If unsure, say N.
|
|
|
|
config SQUASHFS_FRAGMENT_CACHE_SIZE
|
|
int "Number of fragments cached" if SQUASHFS_EMBEDDED
|
|
depends on SQUASHFS
|
|
default "3"
|
|
help
|
|
By default SquashFS caches the last 3 fragments read from
|
|
the filesystem. Increasing this amount may mean SquashFS
|
|
has to re-read fragments less often from disk, at the expense
|
|
of extra system memory. Decreasing this amount will mean
|
|
SquashFS uses less memory at the expense of extra reads from disk.
|
|
|
|
Note there must be at least one cached fragment. Anything
|
|
much more than three will probably not make much difference.
|