Package net.snowflake.ingest.streaming.internal

Class ParquetFlusher

java.lang.Object

net.snowflake.ingest.streaming.internal.ParquetFlusher

All Implemented Interfaces:

Flusher<ParquetChunkData>

public class ParquetFlusher
extends Object
implements Flusher<ParquetChunkData>

Converts ChannelData buffered in RowBuffer to the Parquet format for faster processing.

Nested Class Summary

Nested classes/interfaces inherited from interface net.snowflake.ingest.streaming.internal.Flusher

Flusher.SerializationResult

Constructor Summary

Constructors

Constructor	Description
ParquetFlusher(org.apache.parquet.schema.MessageType schema, long maxChunkSizeInBytes, Optional<Integer> maxRowGroups, Constants.BdecParquetCompression bdecParquetCompression, org.apache.parquet.column.ParquetProperties.WriterVersion parquetWriterVersion, boolean enableDictionaryEncoding, boolean enableIcebergStreaming)	Construct parquet flusher from its schema.

Method Summary

Modifier and Type	Method	Description
Flusher.SerializationResult	serialize(List<net.snowflake.ingest.streaming.internal.ChannelData<ParquetChunkData>> channelsDataPerTable, String filePath, long chunkStartOffset)	Serialize buffered rows into the underlying format.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ParquetFlusher

public ParquetFlusher(org.apache.parquet.schema.MessageType schema,
                      long maxChunkSizeInBytes,
                      Optional<Integer> maxRowGroups,
                      Constants.BdecParquetCompression bdecParquetCompression,
                      org.apache.parquet.column.ParquetProperties.WriterVersion parquetWriterVersion,
                      boolean enableDictionaryEncoding,
                      boolean enableIcebergStreaming)

Construct parquet flusher from its schema.

Method Detail

serialize

public Flusher.SerializationResult serialize(List<net.snowflake.ingest.streaming.internal.ChannelData<ParquetChunkData>> channelsDataPerTable,
                                             String filePath,
                                             long chunkStartOffset)
                                      throws IOException

Description copied from interface: Flusher

Serialize buffered rows into the underlying format.

Specified by:

serialize in interface Flusher<ParquetChunkData>

Parameters:

channelsDataPerTable - buffered rows

filePath - file path

Returns:

Flusher.SerializationResult

Throws:

IOException