org.apache.spark.sql.execution.columnar.impl
ColumnFormatValue
Related Doc:
package impl
<invalid inheritdoc annotation>
<invalid inheritdoc annotation>
This keeps track of whether the buffer is compressed or not.
This keeps track of whether the buffer is compressed or not. In addition it keeps a count of how many times compression was done on the buffer without intervening decompression, and if it exceeds ColumnFormatEntry.MAX_CONSECUTIVE_COMPRESSIONS and no one is using the decompressed buffer, then replace columnBuffer with compressed version.
A negative value indicates that the buffer is not compressible (too small or not enough compression can be achieved), a zero indicates a compressed buffer while a positive count indicates a decompressed buffer and number of times compression was done.
Return the data as a ByteBuffer.
Return the data as a ByteBuffer. Should be invoked only after a retain or getValueRetain call.
Callers of this method should have a corresponding release method for eager release to work else off-heap object may keep around occupying system RAM until the next GC cycle.
Callers of this method should have a corresponding release method for eager release to work else off-heap object may keep around occupying system RAM until the next GC cycle. Callers may decide whether to keep the release method in a finally block to ensure its invocation, or do it only in normal paths because JVM reference collector will eventually clean it in any case.
Calls to this specific class are guaranteed to always return buffers which have position as zero so callers can make simplifying assumptions about the same.
Value object in the column store simply encapsulates binary data as a ByteBuffer. This can be either a direct buffer or a heap buffer depending on the system off-heap configuration. The reason for a separate type is to easily store data off-heap without any major changes to engine otherwise as well as efficiently serialize/deserialize them directly to Oplog/socket.
This class extends SerializedDiskBuffer to avoid a copy when reading/writing from Oplog. Consequently it writes the serialization header itself (typeID + classID + size) into stream as would be written by DataSerializer.writeObject. This helps it avoid additional byte writes when transferring data to the channels.