org.datavec.api.split

Interface InputSplit

All Known Implementing Classes:

BaseInputSplit, CollectionInputSplit, FileSplit, InputStreamInputSplit, ListStringSplit, NumberedFileInputSplit, OutputStreamInputSplit, StringSplit, TransformSplit

public interface InputSplit

An input split. Basically, a list of loadable locations exposed as an iterator.

Author:

Adam Gibson

Method Summary

Modifier and Type	Method and Description
String	addNewLocation() Add a new location with the name generated by this input split/
String	addNewLocation(String location) Add a new location to this input split (this may do anything from updating an in memory location to creating a new file)
void	bootStrapForWrite() Bootstrap this input split for writing.
boolean	canWriteToLocation(URI location) Returns true if the given uri can be written to
long	length() Length of the split
URI[]	locations() Locations of the splits
Iterator<URI>	locationsIterator()
Iterator<String>	locationsPathIterator()
boolean	needsBootstrapForWrite() Returns true if this InputSplit needs bootstrapping for writing.
InputStream	openInputStreamFor(String location) Open an InputStream for the given location.
OutputStream	openOutputStreamFor(String location) Open an OutputStream for the given location.
void	reset() Reset the InputSplit without reinitializing it from scratch.
boolean	resetSupported()
void	updateSplitLocations(boolean reset) Refreshes the split locations if needed in memory.

Method Detail

canWriteToLocation

boolean canWriteToLocation(URI location)

Returns true if the given uri can be written to

Parameters:

location - the location to determine

Returns:

addNewLocation

String addNewLocation()

Add a new location with the name generated by this input split/

addNewLocation

String addNewLocation(String location)

Add a new location to this input split (this may do anything from updating an in memory location to creating a new file)

Parameters:

location - the location to add

updateSplitLocations

void updateSplitLocations(boolean reset)

Refreshes the split locations if needed in memory. (Think a few file gets added)

Parameters:

reset -

needsBootstrapForWrite

boolean needsBootstrapForWrite()

Returns true if this InputSplit needs bootstrapping for writing. A simple example of needing bootstrapping is for FileSplit where there is only a directory existing, but no file to write to

Returns:

true if this input split needs bootstrapping for writing to or not

bootStrapForWrite

void bootStrapForWrite()

Bootstrap this input split for writing. This is for use with RecordWriter

openOutputStreamFor

OutputStream openOutputStreamFor(String location)
                          throws Exception

Open an OutputStream for the given location. Note that the user is responsible for closing the associated output stream.

Parameters:

location - the location to open the output stream for

Returns:

the output input stream

Throws:

Exception

openInputStreamFor

InputStream openInputStreamFor(String location)
                        throws Exception

Open an InputStream for the given location. Note that the user is responsible for closing the associated input stream.

Parameters:

location - the location to open the input stream for

Returns:

the opened input stream

Throws:

Exception

length

long length()

Length of the split

Returns:

locations

URI[] locations()

Locations of the splits

Returns:

locationsIterator

Iterator<URI> locationsIterator()

Returns:

locationsPathIterator

Iterator<String> locationsPathIterator()

Returns:

reset

void reset()

Reset the InputSplit without reinitializing it from scratch. In many cases, this is a no-op. For InputSplits that have randomization: reset should shuffle the order.

resetSupported

boolean resetSupported()

Returns:

True if the reset() method is supported (or is a no-op), false otherwise. If false is returned, reset() may throw an exception