Package org.apache.cassandra.db.streaming

File transfer

See: Description

Interface Summary

Interface	Description
IStreamReader	This is the interface is used by the streaming code read a SSTable stream off a channel.

Class Summary

Class	Description
CassandraCompressedStreamReader	CassandraStreamReader that reads from streamed compressed SSTable
CassandraCompressedStreamWriter	CassandraStreamWriter for compressed SSTable.
CassandraEntireSSTableStreamReader	CassandraEntireSSTableStreamReader reads SSTable off the wire and writes it to disk.
CassandraEntireSSTableStreamWriter	CassandraEntireSSTableStreamWriter streams the entire SSTable to given channel.
CassandraIncomingFile	used to receive the part(or whole) of a SSTable data file.
CassandraOutgoingFile	used to transfer the part(or whole) of a SSTable data file
CassandraStreamHeader
CassandraStreamHeader.Builder
CassandraStreamHeader.CassandraStreamHeaderSerializer
CassandraStreamManager	Implements the streaming interface for the native cassandra storage engine.
CassandraStreamReader	CassandraStreamReader reads from stream and writes to SSTable.
CassandraStreamReader.StreamDeserializer
CassandraStreamReceiver
CassandraStreamWriter	CassandraStreamWriter writes given section of the SSTable to given channel.
ComponentContext	Mutable SSTable components and their hardlinks to avoid concurrent sstable component modification during entire-sstable-streaming.
ComponentManifest	SSTable components and their sizes to be tranfered via entire-sstable-streaming
CompressedInputStream	InputStream which reads compressed chunks from the underlying input stream and deals with decompression and position tracking.
CompressionInfo	Container that carries compression parameters and chunks to decompress data from stream.

Package org.apache.cassandra.db.streaming Description

File transfer

When tranferring whole or subsections of an sstable, only the DATA component is shipped. To that end, there are three "modes" of an sstable transfer that need to be handled somewhat differently: 1) uncompressed sstable - data needs to be read into user space so it can be manipulated: checksum validation, apply stream compression (see next section), and/or TLS encryption. 2) compressed sstable, transferred with SSL/TLS - data needs to be read into user space as that is where the TLS encryption needs to happen. Netty does not allow the pretense of doing zero-copy transfers when TLS is in the pipeline; data must explicitly be pulled into user-space memory for TLS encryption to work. 3) compressed sstable, transferred without SSL/TLS - data can be streamed via zero-copy transfer as the data does not need to be manipulated (it can be sent "as-is").

Compressing the data

We always want to transfer as few bytes as possible of the wire when streaming a file. If the sstable is not already compressed via table compression options, we apply an on-the-fly stream compression to the data. The stream compression format is documented in StreamCompressionSerializer You may be wondering: why implement your own compression scheme? why not use netty's built-in compression codecs, like Lz4FrameEncoder? That makes complete sense if all the sstables to be streamed are non using sstable compression (and obviously you wouldn't use stream compression when the sstables are using sstable compression). The problem is when you have a mix of files, some using sstable compression and some not. You can either: - send the files of one type over one kind of socket, and the others over another socket - send them both over the same socket, but then auto-adjust per each file type. I've opted for the latter to keep socket/channel management simpler and cleaner.