Package org.datavec.api.split
Class FileSplit
- java.lang.Object
-
- org.datavec.api.split.BaseInputSplit
-
- org.datavec.api.split.FileSplit
-
- All Implemented Interfaces:
InputSplit
public class FileSplit extends BaseInputSplit
-
-
Field Summary
Fields Modifier and Type Field Description protected String[]
allowFormat
protected Random
random
protected boolean
randomize
protected boolean
recursive
protected File
rootDir
-
Fields inherited from class org.datavec.api.split.BaseInputSplit
iterationOrder, length, uriStrings
-
-
Constructor Summary
Constructors Modifier Constructor Description FileSplit(File rootDir)
FileSplit(File rootDir, String[] allowFormat)
FileSplit(File rootDir, String[] allowFormat, boolean recursive)
protected
FileSplit(File rootDir, String[] allowFormat, boolean recursive, Random random, boolean runMain)
FileSplit(File rootDir, String[] allowFormat, Random rng)
FileSplit(File rootDir, Random rng)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description String
addNewLocation()
Add a new location with the name generated by this input split/String
addNewLocation(String location)
Add a new location to this input split (this may do anything from updating an in memory location to creating a new file)void
bootStrapForWrite()
Bootstrap this input split for writing.File
getRootDir()
protected void
initialize()
long
length()
Length of the splitboolean
needsBootstrapForWrite()
Returns true if thisInputSplit
needs bootstrapping for writing.InputStream
openInputStreamFor(String location)
Open anInputStream
for the given location.OutputStream
openOutputStreamFor(String location)
Open anOutputStream
for the given location.void
reset()
Reset the InputSplit without reinitializing it from scratch.boolean
resetSupported()
void
updateSplitLocations(boolean reset)
Refreshes the split locations if needed in memory.-
Methods inherited from class org.datavec.api.split.BaseInputSplit
canWriteToLocation, locations, locationsIterator, locationsPathIterator, sample
-
-
-
-
Method Detail
-
initialize
protected void initialize()
-
addNewLocation
public String addNewLocation()
Description copied from interface:InputSplit
Add a new location with the name generated by this input split/- Specified by:
addNewLocation
in interfaceInputSplit
- Overrides:
addNewLocation
in classBaseInputSplit
-
addNewLocation
public String addNewLocation(String location)
Description copied from interface:InputSplit
Add a new location to this input split (this may do anything from updating an in memory location to creating a new file)- Specified by:
addNewLocation
in interfaceInputSplit
- Overrides:
addNewLocation
in classBaseInputSplit
- Parameters:
location
- the location to add
-
updateSplitLocations
public void updateSplitLocations(boolean reset)
Description copied from interface:InputSplit
Refreshes the split locations if needed in memory. (Think a few file gets added)
-
needsBootstrapForWrite
public boolean needsBootstrapForWrite()
Description copied from interface:InputSplit
Returns true if thisInputSplit
needs bootstrapping for writing. A simple example of needing bootstrapping is forFileSplit
where there is only a directory existing, but no file to write to- Returns:
- true if this input split needs bootstrapping for writing to or not
-
bootStrapForWrite
public void bootStrapForWrite()
Description copied from interface:InputSplit
Bootstrap this input split for writing. This is for use withRecordWriter
-
openOutputStreamFor
public OutputStream openOutputStreamFor(String location) throws Exception
Description copied from interface:InputSplit
Open anOutputStream
for the given location. Note that the user is responsible for closing the associated output stream.- Parameters:
location
- the location to open the output stream for- Returns:
- the output input stream
- Throws:
Exception
-
openInputStreamFor
public InputStream openInputStreamFor(String location) throws Exception
Description copied from interface:InputSplit
Open anInputStream
for the given location. Note that the user is responsible for closing the associated input stream.- Parameters:
location
- the location to open the input stream for- Returns:
- the opened input stream
- Throws:
Exception
-
length
public long length()
Description copied from interface:InputSplit
Length of the split- Specified by:
length
in interfaceInputSplit
- Overrides:
length
in classBaseInputSplit
- Returns:
-
reset
public void reset()
Description copied from interface:InputSplit
Reset the InputSplit without reinitializing it from scratch. In many cases, this is a no-op. For InputSplits that have randomization: reset should shuffle the order.
-
resetSupported
public boolean resetSupported()
- Returns:
- True if the reset() method is supported (or is a no-op), false otherwise. If false is returned, reset() may throw an exception
-
getRootDir
public File getRootDir()
-
-