org.apache.hadoop.hbase.io.encoding

Class DiffKeyDeltaEncoder

java.lang.Object

org.apache.hadoop.hbase.io.encoding.DiffKeyDeltaEncoder

All Implemented Interfaces:

DataBlockEncoder

@InterfaceAudience.Private
public class DiffKeyDeltaEncoder
extends Object

Compress using: - store size of common prefix - save column family once, it is same within HFile - use integer compression for key, value and prefix (7-bit encoding) - use bits to avoid duplication key length, value length and type if it same as previous - store in 3 bits length of timestamp field - allow diff in timestamp instead of actual value Format: - 1 byte: flag - 1-5 bytes: key length (only if FLAG_SAME_KEY_LENGTH is not set in flag) - 1-5 bytes: value length (only if FLAG_SAME_VALUE_LENGTH is not set in flag) - 1-5 bytes: prefix length - ... bytes: rest of the row (if prefix length is small enough) - ... bytes: qualifier (or suffix depending on prefix length) - 1-8 bytes: timestamp or diff - 1 byte: type (only if FLAG_SAME_TYPE is not set in the flag) - ... bytes: value

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	BufferedDataBlockEncoder.BufferedEncodedSeeker<STATE extends BufferedDataBlockEncoder.SeekerState>
protected static class	BufferedDataBlockEncoder.SeekerState
protected static class	DiffKeyDeltaEncoder.DiffCompressionState
protected static class	DiffKeyDeltaEncoder.DiffSeekerState

Nested classes/interfaces inherited from interface org.apache.hadoop.hbase.io.encoding.DataBlockEncoder

DataBlockEncoder.EncodedSeeker

Constructor Summary

Constructors

Constructor and Description
DiffKeyDeltaEncoder()

Method Summary

Methods

Modifier and Type	Method and Description
protected void	afterDecodingKeyValue(DataInputStream source, ByteBuffer dest, boolean includesMemstoreTS)
protected void	afterEncodingKeyValue(ByteBuffer in, DataOutputStream out, boolean includesMemstoreTS)
DataBlockEncoder.EncodedSeeker	createSeeker(KeyValue.KVComparator comparator, boolean includesMemstoreTS) Create a HFileBlock seeker which find KeyValues within a block.
ByteBuffer	decodeKeyValues(DataInputStream source, boolean includesMemstoreTS) Decode.
ByteBuffer	decodeKeyValues(DataInputStream source, int allocHeaderLength, int skipLastBytes, boolean includesMemstoreTS) Uncompress.
void	encodeKeyValues(ByteBuffer in, boolean includesMemstoreTS, HFileBlockEncodingContext blkEncodingCtx) Encodes KeyValues.
protected static void	ensureSpace(ByteBuffer out, int length) Asserts that there is at least the given amount of unfilled space remaining in the given buffer.
ByteBuffer	getFirstKeyInBlock(ByteBuffer block) Return first key in block.
void	internalEncodeKeyValues(DataOutputStream out, ByteBuffer in, boolean includesMemstoreTS) Compress KeyValues and write them to output buffer.
HFileBlockDecodingContext	newDataBlockDecodingContext(Compression.Algorithm compressionAlgorithm) Creates an encoder specific decoding context, which will prepare the data before actual decoding
HFileBlockEncodingContext	newDataBlockEncodingContext(Compression.Algorithm compressionAlgorithm, DataBlockEncoding encoding, byte[] header) Creates a encoder specific encoding context
String	toString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

DiffKeyDeltaEncoder

public DiffKeyDeltaEncoder()

Method Detail

internalEncodeKeyValues

public void internalEncodeKeyValues(DataOutputStream out,
                           ByteBuffer in,
                           boolean includesMemstoreTS)
                             throws IOException

Compress KeyValues and write them to output buffer.

Parameters:

out - Where to write compressed data.

in - Source of KeyValue for compression.

includesMemstoreTS - true if including memstore timestamp after every key-value pair

Throws:

IOException - If there is an error writing to output stream.

decodeKeyValues

public ByteBuffer decodeKeyValues(DataInputStream source,
                         int allocHeaderLength,
                         int skipLastBytes,
                         boolean includesMemstoreTS)
                           throws IOException

Description copied from interface: DataBlockEncoder

Uncompress.

Parameters:

source - encoded stream of KeyValues.

allocHeaderLength - allocate this many bytes for the header.

skipLastBytes - Do not copy n last bytes.

includesMemstoreTS - true if including memstore timestamp after every key-value pair

Returns:

Uncompressed block of KeyValues.

Throws:

IOException - If there is an error in source.

getFirstKeyInBlock

public ByteBuffer getFirstKeyInBlock(ByteBuffer block)

Description copied from interface: DataBlockEncoder

Return first key in block. Useful for indexing. Typically does not make a deep copy but returns a buffer wrapping a segment of the actual block's byte array. This is because the first key in block is usually stored unencoded.

Parameters:

block - encoded block we want index, the position will not change

Returns:

First key in block.

toString

public String toString()

Overrides:

toString in class Object

createSeeker

public DataBlockEncoder.EncodedSeeker createSeeker(KeyValue.KVComparator comparator,
                                          boolean includesMemstoreTS)

Description copied from interface: DataBlockEncoder

Create a HFileBlock seeker which find KeyValues within a block.

Parameters:

comparator - what kind of comparison should be used

includesMemstoreTS - true if including memstore timestamp after every key-value pair

Returns:

A newly created seeker.

decodeKeyValues

public ByteBuffer decodeKeyValues(DataInputStream source,
                         boolean includesMemstoreTS)
                           throws IOException

Description copied from interface: DataBlockEncoder

Decode.

Specified by:

decodeKeyValues in interface DataBlockEncoder

Parameters:

source - Compressed stream of KeyValues.

includesMemstoreTS - true if including memstore timestamp after every key-value pair

Returns:

Uncompressed block of KeyValues.

Throws:

IOException - If there is an error in source.

afterEncodingKeyValue

protected final void afterEncodingKeyValue(ByteBuffer in,
                         DataOutputStream out,
                         boolean includesMemstoreTS)

afterDecodingKeyValue

protected final void afterDecodingKeyValue(DataInputStream source,
                         ByteBuffer dest,
                         boolean includesMemstoreTS)

newDataBlockEncodingContext

public HFileBlockEncodingContext newDataBlockEncodingContext(Compression.Algorithm compressionAlgorithm,
                                                    DataBlockEncoding encoding,
                                                    byte[] header)

Description copied from interface: DataBlockEncoder

Creates a encoder specific encoding context

Specified by:

newDataBlockEncodingContext in interface DataBlockEncoder

Parameters:

compressionAlgorithm - compression algorithm used if the final data needs to be compressed

encoding - encoding strategy used

header - header bytes to be written, put a dummy header here if the header is unknown

Returns:

a newly created encoding context

newDataBlockDecodingContext

public HFileBlockDecodingContext newDataBlockDecodingContext(Compression.Algorithm compressionAlgorithm)

Description copied from interface: DataBlockEncoder

Creates an encoder specific decoding context, which will prepare the data before actual decoding

Specified by:

newDataBlockDecodingContext in interface DataBlockEncoder

Parameters:

compressionAlgorithm - compression algorithm used if the data needs to be decompressed

Returns:

a newly created decoding context

encodeKeyValues

public void encodeKeyValues(ByteBuffer in,
                   boolean includesMemstoreTS,
                   HFileBlockEncodingContext blkEncodingCtx)
                     throws IOException

Description copied from interface: DataBlockEncoder

Encodes KeyValues. It will first encode key value pairs, and then optionally do the compression for the encoded data.

Specified by:

encodeKeyValues in interface DataBlockEncoder

Parameters:

in - Source of KeyValue for compression.

includesMemstoreTS - true if including memstore timestamp after every key-value pair

blkEncodingCtx - the encoding context which will contain encoded uncompressed bytes as well as compressed encoded bytes if compression is enabled, and also it will reuse resources across multiple calls.

Throws:

IOException - If there is an error writing to output stream.

ensureSpace

protected static void ensureSpace(ByteBuffer out,
               int length)
                           throws EncoderBufferTooSmallException

Asserts that there is at least the given amount of unfilled space remaining in the given buffer.

Parameters:

out - typically, the buffer we are writing to

length - the required space in the buffer

Throws:

EncoderBufferTooSmallException - If there are no enough bytes.