org.datavec.api.split

Class FileSplit

java.lang.Object

org.datavec.api.split.BaseInputSplit

org.datavec.api.split.FileSplit

All Implemented Interfaces:

InputSplit

public class FileSplit
extends BaseInputSplit

File input split. Splits up a root directory in to files.

Author:

Adam Gibson

Field Summary

Fields

Modifier and Type	Field and Description
protected String[]	allowFormat
protected Random	random
protected boolean	randomize
protected boolean	recursive
protected File	rootDir

Fields inherited from class org.datavec.api.split.BaseInputSplit

iterationOrder, length, uriStrings

Constructor Summary

Constructors

Modifier	Constructor and Description
	FileSplit(File rootDir)
	FileSplit(File rootDir, Random rng)
	FileSplit(File rootDir, String[] allowFormat)
	FileSplit(File rootDir, String[] allowFormat, boolean recursive)
protected	FileSplit(File rootDir, String[] allowFormat, boolean recursive, Random random, boolean runMain)
	FileSplit(File rootDir, String[] allowFormat, Random rng)

Method Summary

Modifier and Type	Method and Description
String	addNewLocation() Add a new location with the name generated by this input split/
String	addNewLocation(String location) Add a new location to this input split (this may do anything from updating an in memory location to creating a new file)
void	bootStrapForWrite() Bootstrap this input split for writing.
File	getRootDir()
protected void	initialize()
long	length() Length of the split
boolean	needsBootstrapForWrite() Returns true if this InputSplit needs bootstrapping for writing.
InputStream	openInputStreamFor(String location) Open an InputStream for the given location.
OutputStream	openOutputStreamFor(String location) Open an OutputStream for the given location.
void	reset() Reset the InputSplit without reinitializing it from scratch.
boolean	resetSupported()
void	updateSplitLocations(boolean reset) Refreshes the split locations if needed in memory.

Methods inherited from class org.datavec.api.split.BaseInputSplit

canWriteToLocation, locations, locationsIterator, locationsPathIterator, sample

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

rootDir

protected File rootDir

allowFormat

protected String[] allowFormat

recursive

protected boolean recursive

random

protected Random random

randomize

protected boolean randomize

Constructor Detail

FileSplit

protected FileSplit(File rootDir,
                    String[] allowFormat,
                    boolean recursive,
                    Random random,
                    boolean runMain)

FileSplit

public FileSplit(File rootDir)

FileSplit

public FileSplit(File rootDir,
                 Random rng)

FileSplit

public FileSplit(File rootDir,
                 String[] allowFormat)

FileSplit

public FileSplit(File rootDir,
                 String[] allowFormat,
                 Random rng)

FileSplit

public FileSplit(File rootDir,
                 String[] allowFormat,
                 boolean recursive)

Method Detail

initialize

protected void initialize()

addNewLocation

public String addNewLocation()

Description copied from interface: InputSplit

Add a new location with the name generated by this input split/

Specified by:

addNewLocation in interface InputSplit

Overrides:

addNewLocation in class BaseInputSplit

addNewLocation

public String addNewLocation(String location)

Description copied from interface: InputSplit

Add a new location to this input split (this may do anything from updating an in memory location to creating a new file)

Specified by:

addNewLocation in interface InputSplit

Overrides:

addNewLocation in class BaseInputSplit

Parameters:

location - the location to add

updateSplitLocations

public void updateSplitLocations(boolean reset)

Description copied from interface: InputSplit

Refreshes the split locations if needed in memory. (Think a few file gets added)

needsBootstrapForWrite

public boolean needsBootstrapForWrite()

Description copied from interface: InputSplit

Returns true if this InputSplit needs bootstrapping for writing. A simple example of needing bootstrapping is for FileSplit where there is only a directory existing, but no file to write to

Returns:

true if this input split needs bootstrapping for writing to or not

bootStrapForWrite

public void bootStrapForWrite()

Description copied from interface: InputSplit

Bootstrap this input split for writing. This is for use with RecordWriter

openOutputStreamFor

public OutputStream openOutputStreamFor(String location)
                                 throws Exception

Description copied from interface: InputSplit

Open an OutputStream for the given location. Note that the user is responsible for closing the associated output stream.

Parameters:

location - the location to open the output stream for

Returns:

the output input stream

Throws:

Exception

openInputStreamFor

public InputStream openInputStreamFor(String location)
                               throws Exception

Description copied from interface: InputSplit

Open an InputStream for the given location. Note that the user is responsible for closing the associated input stream.

Parameters:

location - the location to open the input stream for

Returns:

the opened input stream

Throws:

Exception

length

public long length()

Description copied from interface: InputSplit

Length of the split

Specified by:

length in interface InputSplit

Overrides:

length in class BaseInputSplit

Returns:

reset

public void reset()

Description copied from interface: InputSplit

Reset the InputSplit without reinitializing it from scratch. In many cases, this is a no-op. For InputSplits that have randomization: reset should shuffle the order.

resetSupported

public boolean resetSupported()

Returns:

True if the reset() method is supported (or is a no-op), false otherwise. If false is returned, reset() may throw an exception

getRootDir

public File getRootDir()