org.datavec.api.split

Class StreamInputSplit

java.lang.Object

org.datavec.api.split.StreamInputSplit

All Implemented Interfaces:

InputSplit

public class StreamInputSplit
extends Object
implements InputSplit

StreamInputSplit is a way of specifying input as a bunch of URIs, as well as the way those URIs should be opened. For example, if data was stored remotely (HDFS, S3, etc) you could use StreamInputSplit to load them, doing two things: (a) providing the URIs of the remote data to the constructor
(b) providing a Function<URI,InputStream> that opens an InputStream for the given URI.

Note: supports optional randomization (shuffling of order in which streams are read) via StreamInputSplit(List, Function, Random) by providing a Random instance. If no Random instance is provided, the order will always be according to the order in the provided list of URIs.

Author:

Alex Black

Field Summary

Fields

Modifier and Type	Field and Description
protected int[]	order
protected Random	rng
protected Function<URI,InputStream>	streamCreatorFn
protected List<URI>	uris

Constructor Summary

Constructors

Constructor and Description
StreamInputSplit(List<URI> uris, Function<URI,InputStream> streamCreatorFn) Create a StreamInputSplit with no randomization
StreamInputSplit(List<URI> uris, Function<URI,InputStream> streamCreatorFn, Random rng) Create a StreamInputSplit with optional randomization

Method Summary

Modifier and Type	Method and Description
String	addNewLocation() Add a new location with the name generated by this input split/
String	addNewLocation(String location) Add a new location to this input split (this may do anything from updating an in memory location to creating a new file)
void	bootStrapForWrite() Bootstrap this input split for writing.
boolean	canWriteToLocation(URI location) Returns true if the given uri can be written to
long	length() Length of the split
URI[]	locations() Locations of the splits
Iterator<URI>	locationsIterator()
Iterator<String>	locationsPathIterator()
boolean	needsBootstrapForWrite() Returns true if this InputSplit needs bootstrapping for writing.
InputStream	openInputStreamFor(String location) Open an InputStream for the given location.
OutputStream	openOutputStreamFor(String location) Open an OutputStream for the given location.
void	reset() Reset the InputSplit without reinitializing it from scratch.
boolean	resetSupported()
void	updateSplitLocations(boolean reset) Refreshes the split locations if needed in memory.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

uris

protected List<URI> uris

streamCreatorFn

protected Function<URI,InputStream> streamCreatorFn

rng

protected Random rng

order

protected int[] order

Constructor Detail

StreamInputSplit

public StreamInputSplit(@NonNull
                        List<URI> uris,
                        @NonNull
                        Function<URI,InputStream> streamCreatorFn)

Create a StreamInputSplit with no randomization

Parameters:

uris - The list of URIs to load

streamCreatorFn - The function to be used to create InputStream objects for a given URI.

StreamInputSplit

public StreamInputSplit(@NonNull
                        List<URI> uris,
                        @NonNull
                        Function<URI,InputStream> streamCreatorFn,
                        Random rng)

Create a StreamInputSplit with optional randomization

Parameters:

uris - The list of URIs to load

streamCreatorFn - The function to be used to create InputStream objects for a given URI

rng - Random number generator instance. If non-null: streams will be iterated over in a random order. If null: no randomization (iteration order is according to the URIs list)

Method Detail

canWriteToLocation

public boolean canWriteToLocation(URI location)

Description copied from interface: InputSplit

Returns true if the given uri can be written to

Specified by:

canWriteToLocation in interface InputSplit

Parameters:

location - the location to determine

Returns:

addNewLocation

public String addNewLocation()

Description copied from interface: InputSplit

Add a new location with the name generated by this input split/

Specified by:

addNewLocation in interface InputSplit

addNewLocation

public String addNewLocation(String location)

Description copied from interface: InputSplit

Add a new location to this input split (this may do anything from updating an in memory location to creating a new file)

Specified by:

addNewLocation in interface InputSplit

Parameters:

location - the location to add

updateSplitLocations

public void updateSplitLocations(boolean reset)

Description copied from interface: InputSplit

Refreshes the split locations if needed in memory. (Think a few file gets added)

Specified by:

updateSplitLocations in interface InputSplit

needsBootstrapForWrite

public boolean needsBootstrapForWrite()

Description copied from interface: InputSplit

Returns true if this InputSplit needs bootstrapping for writing. A simple example of needing bootstrapping is for FileSplit where there is only a directory existing, but no file to write to

Specified by:

needsBootstrapForWrite in interface InputSplit

Returns:

true if this input split needs bootstrapping for writing to or not

bootStrapForWrite

public void bootStrapForWrite()

Description copied from interface: InputSplit

Bootstrap this input split for writing. This is for use with RecordWriter

Specified by:

bootStrapForWrite in interface InputSplit

openOutputStreamFor

public OutputStream openOutputStreamFor(String location)
                                 throws Exception

Description copied from interface: InputSplit

Open an OutputStream for the given location. Note that the user is responsible for closing the associated output stream.

Specified by:

openOutputStreamFor in interface InputSplit

Parameters:

location - the location to open the output stream for

Returns:

the output input stream

Throws:

Exception

openInputStreamFor

public InputStream openInputStreamFor(String location)
                               throws Exception

Description copied from interface: InputSplit

Open an InputStream for the given location. Note that the user is responsible for closing the associated input stream.

Specified by:

openInputStreamFor in interface InputSplit

Parameters:

location - the location to open the input stream for

Returns:

the opened input stream

Throws:

Exception

length

public long length()

Description copied from interface: InputSplit

Length of the split

Specified by:

length in interface InputSplit

Returns:

locations

public URI[] locations()

Description copied from interface: InputSplit

Locations of the splits

Specified by:

locations in interface InputSplit

Returns:

locationsIterator

public Iterator<URI> locationsIterator()

Specified by:

locationsIterator in interface InputSplit

Returns:

locationsPathIterator

public Iterator<String> locationsPathIterator()

Specified by:

locationsPathIterator in interface InputSplit

Returns:

reset

public void reset()

Description copied from interface: InputSplit

Reset the InputSplit without reinitializing it from scratch. In many cases, this is a no-op. For InputSplits that have randomization: reset should shuffle the order.

Specified by:

reset in interface InputSplit

resetSupported

public boolean resetSupported()

Specified by:

resetSupported in interface InputSplit

Returns:

True if the reset() method is supported (or is a no-op), false otherwise. If false is returned, reset() may throw an exception