org.datavec.api.split

Class BaseInputSplit

java.lang.Object

org.datavec.api.split.BaseInputSplit

All Implemented Interfaces:

InputSplit

Direct Known Subclasses:

CollectionInputSplit, FileSplit, TransformSplit

public abstract class BaseInputSplit
extends Object
implements InputSplit

Base input split

Author:

Adam Gibson

Field Summary

Fields

Modifier and Type	Field and Description
protected int[]	iterationOrder
protected long	length
protected List<String>	uriStrings

Constructor Summary

Constructors

Constructor and Description
BaseInputSplit()

Method Summary

Modifier and Type	Method and Description
String	addNewLocation() Add a new location with the name generated by this input split/
String	addNewLocation(String location) Add a new location to this input split (this may do anything from updating an in memory location to creating a new file)
boolean	canWriteToLocation(URI location) Returns true if the given uri can be written to
long	length() Length of the split
URI[]	locations() Locations of the splits
Iterator<URI>	locationsIterator()
Iterator<String>	locationsPathIterator()
InputSplit[]	sample(PathFilter pathFilter, double... weights) Samples the locations based on the PathFilter and splits the result into an array of InputSplit objects, with sizes proportional to the weights.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.datavec.api.split.InputSplit

bootStrapForWrite, needsBootstrapForWrite, openInputStreamFor, openOutputStreamFor, reset, resetSupported, updateSplitLocations

Field Detail

uriStrings

protected List<String> uriStrings

iterationOrder

protected int[] iterationOrder

length

protected long length

Constructor Detail

BaseInputSplit

public BaseInputSplit()

Method Detail

canWriteToLocation

public boolean canWriteToLocation(URI location)

Description copied from interface: InputSplit

Returns true if the given uri can be written to

Specified by:

canWriteToLocation in interface InputSplit

Parameters:

location - the location to determine

Returns:

addNewLocation

public String addNewLocation()

Description copied from interface: InputSplit

Add a new location with the name generated by this input split/

Specified by:

addNewLocation in interface InputSplit

addNewLocation

public String addNewLocation(String location)

Description copied from interface: InputSplit

Add a new location to this input split (this may do anything from updating an in memory location to creating a new file)

Specified by:

addNewLocation in interface InputSplit

Parameters:

location - the location to add

locations

public URI[] locations()

Description copied from interface: InputSplit

Locations of the splits

Specified by:

locations in interface InputSplit

Returns:

locationsIterator

public Iterator<URI> locationsIterator()

Specified by:

locationsIterator in interface InputSplit

Returns:

locationsPathIterator

public Iterator<String> locationsPathIterator()

Specified by:

locationsPathIterator in interface InputSplit

Returns:

length

public long length()

Description copied from interface: InputSplit

Length of the split

Specified by:

length in interface InputSplit

Returns:

sample

public InputSplit[] sample(PathFilter pathFilter,
                           double... weights)

Samples the locations based on the PathFilter and splits the result into an array of InputSplit objects, with sizes proportional to the weights.

Parameters:

pathFilter - to modify the locations in some way (null == as is)

weights - to split the locations into multiple InputSplit

Returns:

the sampled locations