Package org.apache.flink.runtime.io.disk.iomanager

Class IOManagerAsync

java.lang.Object

org.apache.flink.runtime.io.disk.iomanager.IOManager

org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync

All Implemented Interfaces:

AutoCloseable, Thread.UncaughtExceptionHandler

public class IOManagerAsync
extends IOManager
implements Thread.UncaughtExceptionHandler

A version of the IOManager that uses asynchronous I/O.

Field Summary

Fields inherited from class org.apache.flink.runtime.io.disk.iomanager.IOManager

executorService, LOG

Constructor Summary

Constructors

Constructor	Description
IOManagerAsync()	Constructs a new asynchronous I/O manager, writing files to the system 's temp directory.
IOManagerAsync(String tempDir)	Constructs a new asynchronous I/O manager, writing file to the given directory.
IOManagerAsync(String[] tempDirs, ExecutorService executorService)	Constructs a new asynchronous I/O manager, writing file round robin across the given directories.
IOManagerAsync(String tempDir, ExecutorService executorService)	Constructs a new asynchronous I/O manager, writing file to the given directory.
IOManagerAsync(ExecutorService executorService)	Constructs a new asynchronous I/O manager, writing files to the system 's temp directory.

Method Summary

Modifier and Type	Method	Description
void	close()	Close method.
BlockChannelReader<org.apache.flink.core.memory.MemorySegment>	createBlockChannelReader(FileIOChannel.ID channelID, LinkedBlockingQueue<org.apache.flink.core.memory.MemorySegment> returnQueue)	Creates a block channel reader that reads blocks from the given channel.
BlockChannelWriter<org.apache.flink.core.memory.MemorySegment>	createBlockChannelWriter(FileIOChannel.ID channelID, LinkedBlockingQueue<org.apache.flink.core.memory.MemorySegment> returnQueue)	Creates a block channel writer that writes to the given channel.
BlockChannelWriterWithCallback<org.apache.flink.core.memory.MemorySegment>	createBlockChannelWriter(FileIOChannel.ID channelID, RequestDoneCallback<org.apache.flink.core.memory.MemorySegment> callback)	Creates a block channel writer that writes to the given channel.
BufferFileReader	createBufferFileReader(FileIOChannel.ID channelID, RequestDoneCallback<Buffer> callback)
BufferFileSegmentReader	createBufferFileSegmentReader(FileIOChannel.ID channelID, RequestDoneCallback<FileSegment> callback)
BufferFileWriter	createBufferFileWriter(FileIOChannel.ID channelID)
BulkBlockChannelReader	createBulkBlockChannelReader(FileIOChannel.ID channelID, List<org.apache.flink.core.memory.MemorySegment> targetSegments, int numBlocks)	Creates a block channel reader that reads all blocks from the given channel directly in one bulk.
void	uncaughtException(Thread t, Throwable e)

Methods inherited from class org.apache.flink.runtime.io.disk.iomanager.IOManager

createBlockChannelReader, createBlockChannelWriter, createChannel, createChannelEnumerator, deleteChannel, getExecutorService, getSpillingDirectories, getSpillingDirectoriesPaths

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

IOManagerAsync

@VisibleForTesting
public IOManagerAsync()

Constructs a new asynchronous I/O manager, writing files to the system 's temp directory.

IOManagerAsync

@VisibleForTesting
public IOManagerAsync(ExecutorService executorService)

Constructs a new asynchronous I/O manager, writing files to the system 's temp directory.

IOManagerAsync

@VisibleForTesting
public IOManagerAsync(String tempDir)

Constructs a new asynchronous I/O manager, writing file to the given directory.

Parameters:

tempDir - The directory to write temporary files to.

IOManagerAsync

@VisibleForTesting
public IOManagerAsync(String tempDir,
                      ExecutorService executorService)

Constructs a new asynchronous I/O manager, writing file to the given directory.

Parameters:

tempDir - The directory to write temporary files to.

IOManagerAsync

public IOManagerAsync(String[] tempDirs,
                      ExecutorService executorService)

Constructs a new asynchronous I/O manager, writing file round robin across the given directories.

Parameters:

tempDirs - The directories to write temporary files to.

Method Detail

close

public void close()
           throws Exception

Close method. Shuts down the reader and writer threads immediately, not waiting for their pending requests to be served. This method waits until the threads have actually ceased their operation.

Specified by:

close in interface AutoCloseable

Overrides:

close in class IOManager

Throws:

Exception

uncaughtException

public void uncaughtException(Thread t,
                              Throwable e)

Specified by:

uncaughtException in interface Thread.UncaughtExceptionHandler

createBlockChannelWriter

public BlockChannelWriter<org.apache.flink.core.memory.MemorySegment> createBlockChannelWriter(FileIOChannel.ID channelID,
                                                                                               LinkedBlockingQueue<org.apache.flink.core.memory.MemorySegment> returnQueue)
                                                                                        throws IOException

Description copied from class: IOManager

Creates a block channel writer that writes to the given channel. The writer adds the written segment to the given queue (to allow for asynchronous implementations).

Specified by:

createBlockChannelWriter in class IOManager

Parameters:

channelID - The descriptor for the channel to write to.

returnQueue - The queue to put the written buffers into.

Returns:

A block channel writer that writes to the given channel.

Throws:

IOException - Thrown, if the channel for the writer could not be opened.

createBlockChannelWriter

public BlockChannelWriterWithCallback<org.apache.flink.core.memory.MemorySegment> createBlockChannelWriter(FileIOChannel.ID channelID,
                                                                                                           RequestDoneCallback<org.apache.flink.core.memory.MemorySegment> callback)
                                                                                                    throws IOException

Description copied from class: IOManager

Creates a block channel writer that writes to the given channel. The writer calls the given callback after the I/O operation has been performed (successfully or unsuccessfully), to allow for asynchronous implementations.

Specified by:

createBlockChannelWriter in class IOManager

Parameters:

channelID - The descriptor for the channel to write to.

callback - The callback to be called for

Returns:

A block channel writer that writes to the given channel.

Throws:

IOException - Thrown, if the channel for the writer could not be opened.

createBlockChannelReader

public BlockChannelReader<org.apache.flink.core.memory.MemorySegment> createBlockChannelReader(FileIOChannel.ID channelID,
                                                                                               LinkedBlockingQueue<org.apache.flink.core.memory.MemorySegment> returnQueue)
                                                                                        throws IOException

Creates a block channel reader that reads blocks from the given channel. The reader reads asynchronously, such that a read request is accepted, carried out at some (close) point in time, and the full segment is pushed to the given queue.

Specified by:

createBlockChannelReader in class IOManager

Parameters:

channelID - The descriptor for the channel to write to.

returnQueue - The queue to put the full buffers into.

Returns:

A block channel reader that reads from the given channel.

Throws:

IOException - Thrown, if the channel for the reader could not be opened.

createBufferFileWriter

public BufferFileWriter createBufferFileWriter(FileIOChannel.ID channelID)
                                        throws IOException

Specified by:

createBufferFileWriter in class IOManager

Throws:

IOException

createBufferFileReader

public BufferFileReader createBufferFileReader(FileIOChannel.ID channelID,
                                               RequestDoneCallback<Buffer> callback)
                                        throws IOException

Specified by:

createBufferFileReader in class IOManager

Throws:

IOException

createBufferFileSegmentReader

public BufferFileSegmentReader createBufferFileSegmentReader(FileIOChannel.ID channelID,
                                                             RequestDoneCallback<FileSegment> callback)
                                                      throws IOException

Specified by:

createBufferFileSegmentReader in class IOManager

Throws:

IOException

createBulkBlockChannelReader

public BulkBlockChannelReader createBulkBlockChannelReader(FileIOChannel.ID channelID,
                                                           List<org.apache.flink.core.memory.MemorySegment> targetSegments,
                                                           int numBlocks)
                                                    throws IOException

Creates a block channel reader that reads all blocks from the given channel directly in one bulk. The reader draws segments to read the blocks into from a supplied list, which must contain as many segments as the channel has blocks. After the reader is done, the list with the full segments can be obtained from the reader.

If a channel is not to be read in one bulk, but in multiple smaller batches, a BlockChannelReader should be used.

Specified by:

createBulkBlockChannelReader in class IOManager

Parameters:

channelID - The descriptor for the channel to write to.

targetSegments - The list to take the segments from into which to read the data.

numBlocks - The number of blocks in the channel to read.

Returns:

A block channel reader that reads from the given channel.

Throws:

IOException - Thrown, if the channel for the reader could not be opened.