Intel® QuickAssist Technology (QAT) QATzip Library

QATzip is a user-space library built on top of the Intel® QuickAssist Technology (QAT) user-space library. It provides extended compression and decompression capabilities by offloading these operations to Intel® QAT Accelerators. QATzip generates data in the standard gzip* format (RFC1952) with extended headers or lz4* blocks and using the lz4* frame format. The resulting data can be decompressed using any compliant gzip* or lz4* implementation. QATzip is optimized to fully leverage the performance benefits of Intel® QuickAssist Technology.

Table of Contents

• Supported Formats
• Features
• QATzip Compression Level Mapping
• Hardware Requirements
• Software Requirements
• Install Instructions
  • Install with the in-tree QAT package
  • Install with the out-of-tree QAT package
  • Install from Docker Image
  • Configuration
  • Enable qzstd
• Test QATzip
• Performance Test With QATzip
• QATzip API Manual
• Limitations
• Licensing
• Legal

Supported Formats

Data Format	Algorithm	QAT Device	Description
QZ_DEFLATE_4B	deflate*	QAT 1.x & QAT 2.0	Data is in DEFLATE* with a 4-byte header.
QZ_DEFLATE_GZIP	deflate*	QAT 1.x & QAT 2.0	Data is in DEFLATE* wrapped by a Gzip* header and footer.
QZ_DEFLATE_GZIP_EXT	deflate*	QAT 1.x & QAT 2.0	Data is in DEFLATE* wrapped by an Intel® QAT Gzip* extension header and footer.
QZ_DEFLATE_RAW	deflate*	QAT 1.x & QAT 2.0	Data is in raw DEFLATE* without any additional header. (Compression only; decompression falls back to software.)
QZ_LZ4	lz4*	QAT 2.0	Data is in LZ4* wrapped by an lz4* frame.
QZ_LZ4S	lz4s*	QAT 2.0	Data is in LZ4S* blocks.

Features

• Accelerated compression and decompression using Intel® QuickAssist Technology, including utilities for file compression and decompression.

• Dynamic memory allocation for zero-copy operations via qzMalloc() and qzFree(), enabling pinned, contiguous buffers for DMA operations.

• Instance over-subscription, allowing multiple threads in the same process to share fewer hardware instances seamlessly.

• Memory allocation backed by huge pages and kernel memory for pinned, contiguous memory access, with fallback to huge pages during kernel memory contention.

• Configurable accelerator device sharing across processes.

• Optional software failover for compression and decompression, ensuring functionality even when system resources are insufficient.

• Streaming interfaces for compression and decompression to improve compression ratios and throughput for piecemeal data submissions.

• Asynchronous interfaces for compression and decompression to achieve lower latency and higher throughput (not supported by the qzip utility). This is particularly beneficial for fewer instances or smaller data packets (below 64KB).

• Latency Sensitive Mode: Designed for high-stress scenarios, this mode offloads part of the workload to the CPU, leveraging Intel's multi-core processors to enhance throughput and reduce latency.

  Note: It is not recommended to enable this mode in low-stress scenarios or for smaller data workloads (below 8KB), as it may result in reduced throughput.

• Adaptive polling mechanisms to reduce CPU usage under stress.

• Support for compressing files and directories into the 7z format using the qzip utility.

• Compatibility with QATzip Gzip* format, which includes a 10-byte header and an 8-byte footer:

  | ID1 (1B) | ID2(0x8B) (1B) | Compression Method (8 = DEFLATE*) (1B) | Flags (1B) | Modification Time (4B) | Extra Flags (1B) | OS (1B) | Deflate Block | CRC32 (4B) | ISIZE (4B) |
  

• Support for QATzip Gzip* extended format, which extends the standard 10-byte Gzip* header by an additional 14 bytes:

  | Length of ext. header (2B) | SI1('Q') (1B) | SI2('Z') (1B) | Length of subheader (2B) | Intel(R) defined field 'Chunksize' (4B) | Intel(R) defined field 'Blocksize' (4B) |
  

• Support for Intel® QATzip 4-byte headers, indicating the length of the compressed block:

  | Intel(R) defined Header (4B) | deflate\* block |
  

• Support for QATzip lz4* format, structured as follows:

  | MagicNb (4B) | FLG (1B) | BD (1B) | CS (8B) | HC (1B) | lz4\* Block | EndMark (4B) |
  

QATzip Compression Level Mapping

The following table shows how standard software zlib* compression levels map to QATzip levels for QAT 1.x and QAT 2.0:

Software zlib* Levels	QAT 1.x Equivalent	QAT 2.0 Equivalent
1 - 4	QATzip Level 1	QATzip Level 1
5	QATzip Level 5	QATzip Level 1
6 - 8	QATzip Level 5	QATzip Level 6
9	QATzip Level 9	QATzip Level 9
10 - 12	Unsupported	QATzip Level 9

Refer to QAT Compression levels that summarizes how QATzip Level translates to hardware-accelerated levels for each QAT generation.

Hardware Requirements

This QATzip library supports compression and decompression offload on the platforms with the following QAT devices acceleration devices:

• Intel® QuickAssist 4xxx Series
• Intel® QuickAssist Adapter 8970
• Intel® QuickAssist Adapter 8960
• Intel® QuickAssist Adapter 8950
• Intel® Atom™ Processor C3000

Software Requirements

• Intel® QuickAssist Technology Driver for Linux* HW v2.0 or v1.7 (Out of Tree), latest from Intel® QuickAssist Technology
• Intel® QATlib for Linux (in-tree) - v25.08 or later
• Zlib: v1.2.7 or later
• LZ4: v1.8.3 or later
• Zstandard (zstd): v1.5.0 or later

Distributions like Fedora 34+, RHEL 8.4+ & 9.0+, CentOS 9 Stream, SUSE SLES15 SP3+, Ubuntu 24.04+ and Debian13+ include the qatzip RPM package in their repositories. This package is built with the QAT_HW qatlib in-tree driver, specifically for 4xxx devices.

Install Instructions

Install with the in-tree QAT package

See the QATlib installation guide for detailed instructions.

From RPM (Fedora 34+, RHEL 8.4+, CentOS 9+, Ubuntu 24.04+, Debian 13+):

# RHEL-based
sudo dnf install -y qatzip qatzip-devel

# Debian-based
sudo apt -y install qatzip libqatzip3 libqatzip-dev

From Source Code:

cd QATzip/
export QZ_ROOT=$(pwd)
./autogen.sh
./configure
make clean && make && sudo make install

Install with the out-of-tree QAT package

Refer to the QAT Installation Guide for detailed setup instructions.

Note: For non-root users, see the non-root user guide. When SVM is disabled, QAT hardware requires DMA-accessible memory. Use QAT USDM component to allocate and free DMA-able memory (see the USDM settings guide).

1. Install dependencies:

   # RHEL-based
   sudo dnf install -y autoconf autoconf-archive automake libtool zlib-devel lz4-devel numactl-devel
   
   # Debian-based
   sudo apt -y install autoconf autoconf-archive automake libtool zlib1g-dev liblz4-dev libnuma-dev

2. Configure:

   cd QATzip/
   export QZ_ROOT=$(pwd)
   export ICP_ROOT=/QAT/PACKAGE/PATH
   ./autogen.sh
   ./configure  # Run ./configure -h for options

3. Build and install:

   make clean && make && sudo make install

Install from Docker Image

• Pre-built image: intel/intel-qatzip
• Build image: Use the Dockerfile Note: This is built with QATlib in-tree driver

Configuration

Note: This section applies only to out-of-tree QAT packages. For in-tree QATlib, see the QATlib configuration guide.

The QATzip library requires a [SHIM] section in its configuration file. Set the environment variable QAT_SECTION_NAME=SHIM or use the provided configuration templates.

Update configuration:

1. Locate example configuration files:

   $QZ_ROOT/config_file/$YOUR_PLATFORM/$CONFIG_TYPE/*.conf
   

   • QAT Devices: 4xxx, c6xx, dh895xcc, c3xxx
   • Types: multiple_process_opt or multiple_thread_opt

2. Copy and apply:

   sudo cp $QZ_ROOT/config_file/$YOUR_PLATFORM/$CONFIG_TYPE/*.conf /etc
   sudo service qat_service restart

For more details, see the QAT Programmer's Guide.

Enable qzstd

To enable the lz4s + postprocessing pipeline, compile qzstd, a sample application supporting ZSTD format compression/decompression.

Prerequisites: Install the zstd static library before proceeding.

Build:

cd $QZ_ROOT
./autogen.sh
./configure --enable-lz4s-postprocessing
make clean && make qzstd

Test:

qzstd $your_input_file

Test QATzip

Verify QATzip functionality using qzip tests below:

qzip -k $your_input_file -O gzipext -A deflate

Compress files to 7z:

qzip -O 7z FILE1 FILE2 FILE3... -o result.7z

Compress directories to 7z:

qzip -O 7z DIR1 DIR2 DIR3... -o result.7z

Decompress 7z archive:

qzip -d result.7z

Decompress directory with gzip/gzipext files:

Use the -R option to recursively decompress gzip/gzipext files within a directory:

qzip -d -R DIR

Performance Test With QATzip

Run the performance test script (uses the qatzip-test app):

cd $QZ_ROOT/test/performance_tests
./run_perf_test.sh

Before testing, update the following in run_perf_test.sh:

• Driver configuration for your QAT device (for OOT driver)
• Thread/process arguments
• max_huge_pages_per_process: Set to at least 6x the number of threads

QATzip API Manual

Please refer to file QATzip-man.pdf under the docs folder and Refer this link for additional QAT Documents

Limitations

• The partitioned internal chunk size of 16 KB, used for QAT hardware DMA, is currently disabled.
• For stream objects:
  • Reset the stream object by calling qzEndStream() before reusing it in another session.
  • Clear the stream object by calling qzEndStream() before clearing the session object with qzTeardownSession() to avoid memory leaks.
  • Ensure the stream length is smaller than strm_buff_sz. If it exceeds this size, QATzip will generate multiple deflate blocks in sequence, with the last block marked as the final block (BFIN set).
  • Performance optimization for the pre-allocation process using a thread-local stream buffer list is planned for a future release.
• For the 7z format:
  • Decompression supports only *.7z archives compressed by qzip.
  • Decompression is limited to software-only processing.
  • Header compression is not supported.
• For LZ4* (de)compression, QATzip supports only a 32 KB history buffer.
• For Zstandard (zstd) format compression, qzstd supports only hw_buffer_sz values less than 128 KB.
• Stream APIs currently support the following:
  • Compression: DEFLATE_GZIP, DEFLATE_GZIP_EXT, and DEFLATE_RAW.
  • Decompression: DEFLATE_GZIP and DEFLATE_GZIP_EXT.
• For DEFLATE_RAW hardware decompression:
  • To offload to hardware, the input data must be a single, complete deflate block with an uncompressed size smaller than the hardware buffer size. Otherwise, it will fall back to software processing. The API's input destination length must be set appropriately by the user.
  • If the input data contains multiple blocks and software fallback is enabled, the operation will default to software processing.
• For DEFLATE_ZLIB hardware compression and decompression:
  • To offload to hardware, the input data must be a single, complete deflate block with a compressed size smaller than or equal to the hardware buffer size. Otherwise, it will fall back to software processing. The API's input destination length must be set appropriately by the user.
  • If the input data contains multiple blocks, only the first complete block will be decompressed by hardware. The user must call the decompression API for subsequent blocks.
• When using the asynchronous API:
  • Instance over-subscription is not supported. The number of threads should not exceed the number of instances, as each thread maintains its own session and instance. Additional threads will offload workloads to software.
• For latency-sensitive mode:
  • Avoid enabling this mode in low-stress scenarios or for smaller data workloads (below 8 KB), as it may result in decreased throughput for QATzip.

Licensing

• Intel® QuickAssist Technology (QAT) QATzip: BSD-3-Clause License. Refer to the LICENSE file in the top-level directory for details.
• Example Intel® QuickAssist Technology Driver Configuration Files: Dual BSD/GPLv2 License. Refer to the file headers and the LICENSE.GPL file in the config_file directory for details.

Legal

Intel, Intel Atom, and Xeon are trademarks of Intel Corporation in the U.S. and/or other countries.

*Other names and brands may be claimed as the property of others.

Copyright © 2016-2026, Intel Corporation. All rights reserved.