This sub-package is in beta testing. Ahead of version 1.0 there may be APIchanges, but these are expected to be minimal, if any.

Hey Checkyourlogs Fans, Today I had an interesting case where a customer called in to let us know that Checkpoints were stuck on their Windows Server 2019 Hyper-V Host. Everything had been working great on this host since it was upgraded from Windows Server 2016 back in November. The only change was we upgraded to. The stringstream is then compressed using LZ4 and transmitted to a remote machine for decompression and deserialization. My issue is that LZ4 will output a memory block of compressed data and an integer specifying the compressed data's size.

This sub-package provides the capability to compress and decompress data usingthe LZ4 frame specification.

The frame specification is recommended for most applications. A key benefit ofusing the frame specification (compared to the block specification) isinteroperability with other implementations.

Low level bindings for full content (de)compression¶

These functions are bindings to the LZ4 Frame API functions for compressing datainto a single frame, and decompressing a full frame of data.

compress(data, compression_level=0, block_size=0, content_checksum=0,block_linked=True, store_size=True, return_bytearray=False)

Compresses data returning the compressed data as a complete frame.

The returned data includes a header and endmark and so is suitablefor writing to a file.

Decompresses a frame of data and returns it as a string of bytes.

Low level bindings for chunked content (de)compression¶

These functions are bindings to the LZ4 Frame API functions allowing piece-wisecompression and decompression. Using them requires managing compression anddecompression contexts manually. An alternative to using these is to use thecontext manager classes described in the section below.

Compression¶

Creates a compression context object.

The compression object is required for compression operations.

compress_begin(context, source_size=0, compression_level=0, block_size=0,content_checksum=0, content_size=1, block_mode=0, frame_type=0,auto_flush=1)

Creates a frame header from a compression context.

Compresses blocks of data and returns the compressed data.

The returned data should be concatenated with the data returned fromlz4.frame.compress_begin and any subsequent calls tolz4.frame.compress_chunk.

Notes

If auto flush is disabled (auto_flush=False when callinglz4.frame.compress_begin) this function may buffer and retainsome or all of the compressed data for future calls tolz4.frame.compress.

Flushes any buffered data held in the compression context.

This flushes any data buffed in the compression context, returning it ascompressed data. The returned data should be appended to the output ofprevious calls to lz4.frame.compress_chunk.

The end_frame argument specifies whether or not the frame should beended. If this is True and end of frame marker will be appended tothe returned data. In this case, if content_checksum was Truewhen calling lz4.frame.compress_begin, then a checksum of the uncompresseddata will also be included in the returned data.

If the end_frame argument is True, the compression context will bereset and can be re-used.

Notes

If end_frame is False but the underlying LZ4 library does not support flushing without ending the frame, a RuntimeError will beraised.

Decompression¶

Creates a decompression context object.

A decompression context is needed for decompression operations.

Resets a decompression context object.

This is useful for recovering from an error or for stopping an unfinisheddecompression and starting a new one with the same context

Decompresses part of a frame of compressed data.

The returned uncompressed data should be concatenated with the datareturned from previous calls to lz4.frame.decompress_chunk

The end of frame indicator is True if the end of the compressedframe has been reached, or False otherwise

Retrieving frame information¶

The following function can be used to retrieve information about a compressed frame.

Given a frame of compressed data, returns information about the frame.

Helper context manager classes¶

These classes, which utilize the low level bindings to the Frame API are moreconvenient to use. They provide context management, and so it is not necessaryto manually create and manage compression and decompression contexts.

Create a LZ4 frame compressor object.

This object can be used to compress data incrementally.

Begin a compression frame.

The returned data contains frame header information. The data returnedfrom subsequent calls to compress() should be concatenated withthis header.

Compresses data and returns it.

This compresses data (a bytes object), returning a bytes orbytearray object containing compressed data the input.

If auto_flush has been set to False, some of data may bebuffered internally, for use in later calls toLZ4FrameCompressor.compress() and LZ4FrameCompressor.flush().

The returned data should be concatenated with the output of anyprevious calls to compress() and a single call tocompress_begin().

Finish the compression process.

This returns a bytes or bytearray object containing any datastored in the compressor’s internal buffers and a frame footer.

The LZ4FrameCompressor instance may be re-used after this method hasbeen called to create a new frame of compressed data.

Reset the LZ4FrameCompressor instance.

This allows the LZ4FrameCompression instance to be re-used after anerror.

Create a LZ4 frame decompressor object.

This can be used to decompress data incrementally.

For a more convenient way of decompressing an entire compressed frame atonce, see lz4.frame.decompress().

True if the end-of-stream marker has been reached.False otherwise.

Data found after the end of the compressed stream.Before the end of the frame is reached, this will be b'.

False if the decompress() method canprovide more decompressed data before requiring new uncompressedinput. True otherwise.

Decompresses part or all of an LZ4 frame of compressed data.

The returned data should be concatenated with the output of anyprevious calls to decompress().

If max_length is non-negative, returns at most max_length bytesof decompressed data. If this limit is reached and further output canbe produced, the needs_input attribute will be set to False. Inthis case, the next call to decompress() may provide data asb' to obtain more of the output. In all cases, any unconsumed datafrom previous calls will be prepended to the input data.

If all of the input data was decompressed and returned (eitherbecause this was less than max_length bytes, or becausemax_length was negative), the needs_input attribute will be setto True.

If an end of frame marker is encountered in the data duringdecompression, decompression will stop at the end of the frame, and anydata after the end of frame is available from the unused_dataattribute. In this case, the LZ4FrameDecompressor instance is resetand can be used for further decompression.

Reset the decompressor state.

This is useful after an error occurs, allowing re-use of the instance.

Reading and writing compressed files¶

These provide capability for reading and writing of files using LZ4 compressedframes. These are designed to be drop in replacements for the LZMA, BZ2 and Gzipequivalent functionalities in the Python standard library.

Open an LZ4Frame-compressed file in binary or text mode.

filename can be either an actual file name (given as a str, bytes, orPathLike object), in which case the named file is opened, or it can be anexisting file object to read from or write to.

The mode argument can be 'r', 'rb' (default), 'w','wb', 'x', 'xb', 'a', or 'ab' for binary mode, or'rt', 'wt', 'xt', or 'at' for text mode.

For binary mode, this function is equivalent to the LZ4FrameFileconstructor: LZ4FrameFile(filename,mode,..).

For text mode, an LZ4FrameFile object is created, and wrapped in anio.TextIOWrapper instance with the specified encoding, error handlingbehavior, and line ending(s).

A file object providing transparent LZ4F (de)compression.

An LZ4FFile can act as a wrapper for an existing file object, or referdirectly to a named file on disk.

Note that LZ4FFile provides a binary file interface - data read isreturned as bytes, and data to be written must be given as bytes.

When opening a file for writing, the settings used by the compressor can bespecified. The underlying compressor object islz4.frame.LZ4FrameCompressor. See the docstrings for that class fordetails on compression options.

Flush and close the file.

May be called more than once without error. Once the file isclosed, any other operation on it will raise a ValueError.

Returns True if this file is closed.

Return the file descriptor for the underlying file.

Return buffered data without advancing the file position.

Always returns at least one byte of data, unless at EOF. The exactnumber of bytes returned is unspecified.

Read up to size uncompressed bytes from the file.

If size is negative or omitted, read until EOF is reached.Returns b' if the file is already at EOF.

Read up to size uncompressed bytes.

This method tries to avoid making multiple reads from the underlyingstream.

This method reads up to a buffer’s worth of data if size isnegative.

Returns b' if the file is at EOF.

Return whether the file was opened for reading.

Read a line of uncompressed bytes from the file. Intel gma 3600 windows 8 64 bit drivers.

The terminating newline (if present) is retained. If size isnon-negative, no more than size bytes will be read (in which case theline may be incomplete). Returns b’’ if already at EOF.

Change the file position.

The new position is specified by offset, relative to the positionindicated by whence. Possible values for whence are:

• io.SEEK_SET or 0: start of stream (default): offset must not benegative
• io.SEEK_CUR or 1: current stream position
• io.SEEK_END or 2: end of stream; offset must not be positive

Returns the new file position.

Note that seeking is emulated, so depending on the parameters, thisoperation may be extremely slow.

Return whether the file supports seeking.

Return the current file position.

Return whether the file was opened for writing.

Write a bytes object to the file.

Returns the number of uncompressed bytes written, which is alwayslen(data). Note that due to buffering, the file on disk may notreflect the data written until close() is called.

Module attributes¶

A number of module attributes are defined for convenience. These are detailed below.

Compression level¶

The following module attributes can be used when setting thecompression_level argument.

Specifier for the minimum compression level.

Specifying compression_level=lz4.frame.COMPRESSIONLEVEL_MIN willinstruct the LZ4 library to use a compression level of 0

Specifier for the minimum compression level for high compression mode.

Specifying compression_level=lz4.frame.COMPRESSIONLEVEL_MINHC willinstruct the LZ4 library to use a compression level of 3, the minimum for thehigh compression mode.

Specifier for the maximum compression level.

Specifying compression_level=lz4.frame.COMPRESSIONLEVEL_MAX willinstruct the LZ4 library to use a compression level of 16, the highestcompression level available.

Block size¶

The following attributes can be used when setting the block_size argument.

Specifier for the default block size.

Specifying block_size=lz4.frame.BLOCKSIZE_DEFAULT will instruct the LZ4library to use the default maximum blocksize. This is currently equivalent tolz4.frame.BLOCKSIZE_MAX64KB

Specifier for a maximum block size of 64 kB.

Specifying block_size=lz4.frame.BLOCKSIZE_MAX64KB will instruct the LZ4library to create blocks containing a maximum of 64 kB of uncompressed data.

Specifier for a maximum block size of 256 kB.

Specifying block_size=lz4.frame.BLOCKSIZE_MAX256KB will instruct the LZ4library to create blocks containing a maximum of 256 kB of uncompressed data.

Specifier for a maximum block size of 1 MB.

Specifying block_size=lz4.frame.BLOCKSIZE_MAX1MB will instruct the LZ4library to create blocks containing a maximum of 1 MB of uncompressed data.

Specifier for a maximum block size of 4 MB.

Specifying block_size=lz4.frame.BLOCKSIZE_MAX4MB will instruct the LZ4library to create blocks containing a maximum of 4 MB of uncompressed data.

LZ4 is a lossless data compressionalgorithm that is focused on compression and decompression speed. Cdk software install agent download. It belongs to the LZ77 family of byte-oriented compression schemes.

Features[edit]

The algorithm gives a slightly worse compression ratio than the LZO algorithm – which in turn is worse than algorithms like DEFLATE. However, compression speeds are similar to LZO and several times faster than DEFLATE, while decompression speeds can be significantly higher than LZO.^[2]

Design[edit]

The LZ4 algorithm represents the data as a series of sequences. Each sequence begins with a one-byte token that is broken into two 4-bit fields. The first field represents the number of literal bytes that are to be copied to the output. The second field represents the number of bytes to copy from the already decoded output buffer (with 0 representing the minimum match length of 4 bytes). A value of 15 in either of the bitfields indicates that the length is larger and there is an extra byte of data that is to be added to the length. A value of 255 in these extra bytes indicates that yet another byte to be added. Hence arbitrary lengths are represented by a series of extra bytes containing the value 255. The string of literals comes after the token and any extra bytes needed to indicate string length. This is followed by an offset that indicates how far back in the output buffer to begin copying. The extra bytes (if any) of the match-length come at the end of the sequence.^[3][4]

Compression can be carried out in a stream or in blocks. Higher compression ratios can be achieved by investing more effort in finding the best matches. This results in both a smaller output and faster decompression.

Implementation[edit]

The reference implementation in C by Yann Collet is licensed under a BSD license. There are ports and bindings in various languages like Java, C#, Python etc.^[5] Databases like Hadoop use this algorithm for fast compression. LZ4 was also implemented natively in the Linux kernel 3.11.^[6] The FreeBSD, Illumos, ZFS on Linux, and ZFS-OSX implementations of the ZFS filesystem support the LZ4 algorithm for on-the-fly compression.^{[7][8][9][10]} Linux supports LZ4 for SquashFS since 3.19-rc1.^[11] LZ4 is also implemented in newer Zstd archiver by Yann Collet.

References[edit]

1. ^'LZ4 v1.9.2'. Github. 30 August 2019. Retrieved 30 August 2019.
2. ^Michael Larabel (2013-01-28). 'Support For Compressing The Linux Kernel With LZ4'. Phoronix. Retrieved 2015-08-28.
3. ^Yann Collet (2011-05-26). 'RealTime Data Compression'. Retrieved 2015-08-28.
4. ^ticki (2016-10-25). 'How LZ4 works'. Retrieved 2017-06-29.
5. ^Extremely Fast Compression algorithm http://www.lz4.org on GitHub
6. ^Jonathan Corbet (2013-07-19). 'Kernel development'. LWN.net. Retrieved 2015-08-28.
7. ^'FreeBSD 9.2-RELEASE Release Notes'. FreeBSD. 2013-11-13. Retrieved 2015-08-28.
8. ^'LZ4 Compression'. illumos. Retrieved 2015-08-28.
9. ^Illumos #3035 LZ4 compression support in ZFS and GRUB on GitHub
10. ^'Features: lz4 compression'. OpenZFS. Retrieved 2015-08-28.
11. ^Phillip Lougher (2014-11-27). 'Squashfs: Add LZ4 compression configuration option'. Retrieved 2015-08-28.

External links[edit]

• Official website