org.deeplearning4j.datasets

Class DataSet

java.lang.Object

org.deeplearning4j.berkeley.Pair<org.jblas.DoubleMatrix,org.jblas.DoubleMatrix>

org.deeplearning4j.datasets.DataSet

All Implemented Interfaces:

Serializable, Iterable<DataSet>, Persistable

public class DataSet
extends Pair<org.jblas.DoubleMatrix,org.jblas.DoubleMatrix>
implements Persistable, Iterable<DataSet>

A data set (example/outcome pairs) The outcomes are specifically for neural network encoding such that any labels that are considered true are 1s. The rest are zeros.

Author:

Adam Gibson

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.deeplearning4j.berkeley.Pair

Pair.DefaultLexicographicPairComparator<F extends Comparable<F>,S extends Comparable<S>>, Pair.FirstComparator<S extends Comparable<? super S>,T>, Pair.LexicographicPairComparator<F,S>, Pair.ReverseFirstComparator<S extends Comparable<? super S>,T>, Pair.ReverseSecondComparator<S,T extends Comparable<? super T>>, Pair.SecondComparator<S,T extends Comparable<? super T>>

Constructor Summary

Constructors

Constructor and Description
DataSet(org.jblas.DoubleMatrix first, org.jblas.DoubleMatrix second)
DataSet(Pair<org.jblas.DoubleMatrix,org.jblas.DoubleMatrix> pair)

Method Summary

Methods

Modifier and Type	Method and Description
void	addRow(DataSet d, int i)
List<DataSet>	asList()
List<List<DataSet>>	batchBy(int num)
List<List<DataSet>>	batchByNumLabels()
DataSet	copy()
List<DataSet>	dataSetBatches(int num)
static DataSet	empty()
org.jblas.DoubleMatrix	exampleMaxs()
org.jblas.DoubleMatrix	exampleMeans()
org.jblas.DoubleMatrix	exampleSums()
DataSet	get(int i)
Iterator<DataSet>	iterator()
DataSetIterator	iterator(int batches)
static DataSet	load(File path)
void	load(InputStream is)
static void	main(String[] args)
static DataSet	merge(List<DataSet> data)
int	numExamples()
int	numInputs()
int	numOutcomes()
int	outcome()
void	roundToTheNearest(int roundTo)
DataSet	sample(int numSamples) Sample without replacement and a random rng
DataSet	sample(int numSamples, boolean withReplacement) Sample a dataset numSamples times
DataSet	sample(int numSamples, org.apache.commons.math3.random.RandomGenerator rng) Sample without replacement
DataSet	sample(int numSamples, org.apache.commons.math3.random.RandomGenerator rng, boolean withReplacement) Sample a dataset
void	saveTo(File file, boolean binary)
List<List<DataSet>>	sortAndBatchByNumLabels() Sorts the dataset by label: Splits the data set such that examples are sorted by their labels.
void	sortByLabel() Organizes the dataset to minimize sampling error while still allowing efficient batching.
Pair<DataSet,DataSet>	splitTestAndTrain(int numHoldout)
String	toString()
void	validate()
void	write(OutputStream os)

Methods inherited from class org.deeplearning4j.berkeley.Pair

equals, getFirst, getSecond, hashCode, makePair, newPair, reverse, setFirst, setSecond

Methods inherited from class java.lang.Object

clone, finalize, getClass, notify, notifyAll, wait, wait, wait

Constructor Detail

DataSet

public DataSet(Pair<org.jblas.DoubleMatrix,org.jblas.DoubleMatrix> pair)

DataSet

public DataSet(org.jblas.DoubleMatrix first,
       org.jblas.DoubleMatrix second)

Method Detail

iterator

public DataSetIterator iterator(int batches)

copy

public DataSet copy()

empty

public static DataSet empty()

merge

public static DataSet merge(List<DataSet> data)

numInputs

public int numInputs()

validate

public void validate()

outcome

public int outcome()

get

public DataSet get(int i)

batchBy

public List<List<DataSet>> batchBy(int num)

dataSetBatches

public List<DataSet> dataSetBatches(int num)

sortAndBatchByNumLabels

public List<List<DataSet>> sortAndBatchByNumLabels()

Sorts the dataset by label: Splits the data set such that examples are sorted by their labels. A ten label dataset would produce lists with batches like the following: x1 y = 1 x2 y = 2 ... x10 y = 10

Returns:

a list of data sets partitioned by outcomes

batchByNumLabels

public List<List<DataSet>> batchByNumLabels()

asList

public List<DataSet> asList()

splitTestAndTrain

public Pair<DataSet,DataSet> splitTestAndTrain(int numHoldout)

sortByLabel

public void sortByLabel()

Organizes the dataset to minimize sampling error while still allowing efficient batching.

addRow

public void addRow(DataSet d,
          int i)

exampleSums

public org.jblas.DoubleMatrix exampleSums()

exampleMaxs

public org.jblas.DoubleMatrix exampleMaxs()

exampleMeans

public org.jblas.DoubleMatrix exampleMeans()

saveTo

public void saveTo(File file,
          boolean binary)
            throws IOException

Throws:

IOException

load

public static DataSet load(File path)
                    throws IOException

Throws:

IOException

sample

public DataSet sample(int numSamples)

Sample without replacement and a random rng

Parameters:

numSamples - the number of samples to get

Returns:

a sample data set without replacement

sample

public DataSet sample(int numSamples,
             org.apache.commons.math3.random.RandomGenerator rng)

Sample without replacement

Parameters:

numSamples - the number of samples to get

rng - the rng to use

Returns:

the sampled dataset without replacement

sample

public DataSet sample(int numSamples,
             boolean withReplacement)

Sample a dataset numSamples times

Parameters:

numSamples - the number of samples to get

withReplacement - the rng to use

Returns:

the sampled dataset without replacement

sample

public DataSet sample(int numSamples,
             org.apache.commons.math3.random.RandomGenerator rng,
             boolean withReplacement)

Sample a dataset

Parameters:

numSamples - the number of samples to get

rng - the rng to use

withReplacement - whether to allow duplicates (only tracked by example row number)

Returns:

the sample dataset

roundToTheNearest

public void roundToTheNearest(int roundTo)

numOutcomes

public int numOutcomes()

numExamples

public int numExamples()

toString

public String toString()

Overrides:

toString in class Pair<org.jblas.DoubleMatrix,org.jblas.DoubleMatrix>

main

public static void main(String[] args)
                 throws IOException

Throws:

IOException

write

public void write(OutputStream os)

Specified by:

write in interface Persistable

load

public void load(InputStream is)

Specified by:

load in interface Persistable

iterator

public Iterator<DataSet> iterator()

Specified by:

iterator in interface Iterable<DataSet>