Package net.snowflake.ingest.streaming.internal

Class ParquetRowBuffer

java.lang.Object

net.snowflake.ingest.streaming.internal.ParquetRowBuffer

public class ParquetRowBuffer
extends Object

The buffer in the Streaming Ingest channel that holds the un-flushed rows, these rows will be converted to Parquet format for faster processing

Field Summary

Fields

Modifier and Type	Field	Description
protected Map<String,net.snowflake.ingest.streaming.internal.ParquetColumn>	fieldIndex

Method Summary

Modifier and Type	Method	Description
void	close(String name)	Close the row buffer and release allocated memory for the channel.
Flusher<ParquetChunkData>	createFlusher()
net.snowflake.ingest.streaming.internal.ChannelData<T>	flush()	Flush the data in the row buffer by taking the ownership of the old vectors and pass all the required info back to the flush service to build the blob
float	getSize()	Get the current buffer size
InsertValidationResponse	insertRows(Iterable<Map<String,Object>> rows, String startOffsetToken, String endOffsetToken)	Insert a batch of rows into the row buffer
void	setupSchema(List<net.snowflake.ingest.streaming.internal.ColumnMetadata> columns)	Set up the parquet schema.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

fieldIndex

protected final Map<String,net.snowflake.ingest.streaming.internal.ParquetColumn> fieldIndex

Method Detail

setupSchema

public void setupSchema(List<net.snowflake.ingest.streaming.internal.ColumnMetadata> columns)

Set up the parquet schema.

Parameters:

columns - top level columns list of column metadata

createFlusher

public Flusher<ParquetChunkData> createFlusher()

getSize

public float getSize()

Get the current buffer size

Returns:

the current buffer size

insertRows

public InsertValidationResponse insertRows(Iterable<Map<String,Object>> rows,
                                           String startOffsetToken,
                                           String endOffsetToken)

Insert a batch of rows into the row buffer

Parameters:

rows - input row

startOffsetToken - start offset token of the batch

endOffsetToken - offset token of the latest row in the batch

Returns:

insert response that possibly contains errors because of insertion failures

flush

public net.snowflake.ingest.streaming.internal.ChannelData<T> flush()

Flush the data in the row buffer by taking the ownership of the old vectors and pass all the required info back to the flush service to build the blob

Returns:

A ChannelData object that contains the info needed by the flush service to build a blob

close

public void close(String name)

Close the row buffer and release allocated memory for the channel.