zstd – A Fast Data Compression Algorithm Used By Facebook

Zstandard (also known as zstd) is a free open source, fast real-time data compression program with better compression ratios, developed by Facebook. It is a lossless compression algorithm written in C (there is a re-implementation in Java) – its thus a native Linux program.

Read Also: 10 7zip (Data Comperssion) Command Examples in Linux

When required, it can trade compression speed for stronger compression ratios (compression speed vs compression ratio trade-off can be configured by small increments), vice versa. It has a special mode for small data compression, known as dictionary compression, and can build dictionaries from any sample set provided. It comes with a command line utility for creating and decoding .zst, .gz, .xz and .lz4 files.

Importantly, Zstandard has a rich collection of APIs, supports almost all popular programming languages including Python, Java, JavaScript, Nodejs, Perl, Ruby, C#, Go, Rust, PHP, Switft, and lots more.

It is actively used to compress large volumes of data in multiple formats and use cases in Facebook; services such as Amazon Redshift data warehousing; databases such as Hadoop and Redis; the Tor network and many other applications including games.

The following results are obtained by doing several fast compression algorithms tests on a server running Linux Debian using lzbench, an open-source in-memory benchmark tool.

Zstandard Compression Testing

How to Install Zstandard Compression Tool in Linux

To install Zstandard on a Linux distribution, you need to compile it from sources, but before that first you need to install the necessary development tools on your system using your distribution package manager as shown.

$ sudo apt update && sudo apt install build-essential		#Ubuntu/Debian
# yum group install "Development Tools" 			#CentOS/REHL
# dnf groupinstall "C Development Tools and Libraries"		#Fedora 22+

Once all the needed development tools installed, now you can download the source package, move into the local repo directory, build the binary and install it as shown.

$ cd ~/Downloads
$ git clone https://github.com/facebook/zstd.git
$ cd zstd
$ make
$ sudo make install 

Once Zstandard installed, now we can move further to learn some basic usage of Zstd command examples in the following section.

Learn 10 Zstd Command Usage Examples in Linux

Zstd’s command line syntax is generally similar to that of gzip and xz tools, with a few differences.

1. To create a .zst compression file, simply provide a filename to compress it or use the -z flag also means compress, which is the default action.

$ zstd etcher-1.3.1-x86_64.AppImage 
OR
$ zstd -z etcher-1.3.1-x86_64.AppImage 

2. To decompress a .zst compression file, use the -d flag or the unzstd utility as shown.

$ zstd -d etcher-1.3.1-x86_64.AppImage 
OR
$ unzstd etcher-1.3.1-x86_64.AppImage 

3. To remove source file after an operation, by default, the source file is not deleted after successful compression or decompression, to delete it, use the --rm option.

$ ls etcher-1.3.1-x86_64.AppImage
$ zstd --rm  etcher-1.3.1-x86_64.AppImage
$ ls etcher-1.3.1-x86_64.AppImage

4. To set a compression level, zstd has a number of operation modifiers, for instance you can specify a compression level as -6(a number 1-19, default is 3) as shown.

$ zstd -6 --rm etcher-1.3.1-x86_64.AppImage

5. To set a compression speed, zstd has a compression speed ratio 1-10, the default compression speed is 1. You can trade compression ratio for compression speed with the --fast option, the higher the number the faster the compression speed.

$ zstd --fast=10 etcher-1.3.1-x86_64.AppImage

6. To display information about a compressed file, use the -l flag, which is used to display information about a compressed file, for example.

$ zstd -l etcher-1.3.1-x86_64.AppImage.zst

7. To test the integrity of a compressed files, use the -t flag as shown.

$ zstd -t etcher-1.3.1-x86_64.AppImage.zst

8. To enable verbose mode, use the -v option.

$ zstd -v -5 etcher-1.3.1-x86_64.AppImage

9. To use other file compression or decompression formats such as gzip, xz, lzma, and lz4, using the --format=FORMAT as shown.

$ zstd -v --format=gzip etcher-1.3.1-x86_64.AppImage
$ zstd -v --format=xz  etcher-1.3.1-x86_64.AppImage

10. To set a zstd process priority to real-time, you can use the option –priority=rt as shown.

$zstd --priority=rt etcher-1.3.1-x86_64.AppImage

The -r flag instructs zstd to operate recursively on dictionaries. You can find lots of useful and advanced options, how to read or create dictionaries by consulting the zstd man page.

$ man zstd

Zstandard Github Repository: https://github.com/facebook/zstd

Zstandard is a fast real-time, lossless data compression algorithm and compression tool which offers high compression ratios. Try it out and share your thoughts about it or ask questions via the feedback form below.