Package org.nd4j.linalg.dataset.api

Class DataSetUtil

java.lang.Object

org.nd4j.linalg.dataset.api.DataSetUtil

public class DataSetUtil
extends Object

Constructor Summary

Constructors

Constructor	Description
DataSetUtil()

Method Summary

Modifier and Type	Method	Description
static Pair<INDArray,INDArray>	merge2d(@NonNull INDArray[][] arrays, INDArray[][] masks, int inOutIdx)	Merge the specified 2d arrays and masks.
static Pair<INDArray,INDArray>	merge2d(INDArray[] arrays, INDArray[] masks)	Merge the specified 2d arrays and masks.
static Pair<INDArray,INDArray>	merge4d(INDArray[][] arrays, INDArray[][] masks, int inOutIdx)	Merge the specified 4d arrays and masks.
static Pair<INDArray,INDArray>	merge4d(INDArray[] arrays, INDArray[] masks)	Merge the specified 4d arrays and masks.
static Pair<INDArray[],INDArray[]>	mergeFeatures(@NonNull INDArray[][] featuresToMerge, INDArray[][] featureMasksToMerge)	Merge all of the features arrays into one minibatch.
static Pair<INDArray,INDArray>	mergeFeatures(@NonNull INDArray[] featuresToMerge, INDArray[] featureMasksToMerge)	Merge the specified features and mask arrays (i.e., concatenate the examples)
static Pair<INDArray,INDArray>	mergeFeatures(INDArray[][] featuresToMerge, INDArray[][] featureMasksToMerge, int inOutIdx)	Extract out the specified column, and merge the specified features and mask arrays (i.e., concatenate the examples)
static Pair<INDArray,INDArray>	mergeLabels(@NonNull INDArray[][] labelsToMerge, INDArray[][] labelMasksToMerge, int inOutIdx)	Extract out the specified column, and merge the specified label and label mask arrays (i.e., concatenate the examples)
static Pair<INDArray,INDArray>	mergeLabels(INDArray[] labelsToMerge, INDArray[] labelMasksToMerge)	Merge the specified labels and label mask arrays (i.e., concatenate the examples)
static INDArray	mergeMasks2d(long[] outShape, INDArray[] arrays, INDArray[] masks)
static INDArray	mergeMasks4d(INDArray[] featuresOrLabels, INDArray[] masks)
static INDArray	mergePerOutputMasks2d(long[] outShape, INDArray[][] arrays, INDArray[][] masks, int inOutIdx)
static INDArray	mergePerOutputMasks2d(long[] outShape, INDArray[] arrays, INDArray[] masks)	Deprecated. USe mergeMasks2d(long[], INDArray[], INDArray[])
static Pair<INDArray,INDArray>	mergeTimeSeries(INDArray[][] arrays, INDArray[][] masks, int inOutIdx)	Merge the specified time series (3d) arrays and masks.
static Pair<INDArray,INDArray>	mergeTimeSeries(INDArray[] arrays, INDArray[] masks)	Merge the specified time series (3d) arrays and masks.
static void	setMaskedValuesToZero(INDArray data, INDArray mask)
static INDArray	tailor2d(@NonNull INDArray data, INDArray mask)
static INDArray	tailor2d(@NonNull DataSet dataSet, boolean areFeatures)
static INDArray	tailor3d2d(@NonNull INDArray data, INDArray mask)
static INDArray	tailor3d2d(DataSet dataset, boolean areFeatures)	Deprecated.
static INDArray	tailor4d2d(@NonNull INDArray data)
static INDArray	tailor4d2d(DataSet dataset, boolean areFeatures)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DataSetUtil

public DataSetUtil()

Method Detail

tailor2d

public static INDArray tailor2d(@NonNull
                                @NonNull DataSet dataSet,
                                boolean areFeatures)

tailor2d

public static INDArray tailor2d(@NonNull
                                @NonNull INDArray data,
                                INDArray mask)

tailor3d2d

public static INDArray tailor3d2d(DataSet dataset,
                                  boolean areFeatures)

Deprecated.

tailor3d2d

public static INDArray tailor3d2d(@NonNull
                                  @NonNull INDArray data,
                                  INDArray mask)

tailor4d2d

public static INDArray tailor4d2d(DataSet dataset,
                                  boolean areFeatures)

tailor4d2d

public static INDArray tailor4d2d(@NonNull
                                  @NonNull INDArray data)

setMaskedValuesToZero

public static void setMaskedValuesToZero(INDArray data,
                                         INDArray mask)

mergeFeatures

public static Pair<INDArray[],INDArray[]> mergeFeatures(@NonNull
                                                              @NonNull INDArray[][] featuresToMerge,
                                                              INDArray[][] featureMasksToMerge)

Merge all of the features arrays into one minibatch.

Parameters:

featuresToMerge - features to merge. Note that first index is the input array (example) index, the second index is the input array. Thus to merge 10 examples with 3 input arrays each, featuresToMerge will be indexed like featuresToMerge[0..9][0..2]

featureMasksToMerge - May be null. If non-null: feature masks to merge

Returns:

Merged features, and feature masks. Note that feature masks may be added automatically, if required - even if no feature masks were present originally

mergeFeatures

public static Pair<INDArray,INDArray> mergeFeatures(@NonNull
                                                          @NonNull INDArray[] featuresToMerge,
                                                          INDArray[] featureMasksToMerge)

Merge the specified features and mask arrays (i.e., concatenate the examples)

Parameters:

featuresToMerge - Features to merge

featureMasksToMerge - Mask arrays to merge. May be null

Returns:

Merged features and mask. Mask may be null

mergeFeatures

public static Pair<INDArray,INDArray> mergeFeatures(INDArray[][] featuresToMerge,
                                                          INDArray[][] featureMasksToMerge,
                                                          int inOutIdx)

Extract out the specified column, and merge the specified features and mask arrays (i.e., concatenate the examples)

Parameters:

featuresToMerge - Features to merge. Will use featuresToMerge[all][inOutIdx]

featureMasksToMerge - Mask arrays to merge. May be null

Returns:

Merged features and mask. Mask may be null

mergeLabels

public static Pair<INDArray,INDArray> mergeLabels(INDArray[] labelsToMerge,
                                                        INDArray[] labelMasksToMerge)

Merge the specified labels and label mask arrays (i.e., concatenate the examples)

Parameters:

labelsToMerge - Features to merge

labelMasksToMerge - Mask arrays to merge. May be null

Returns:

Merged features and mask. Mask may be null

mergeLabels

public static Pair<INDArray,INDArray> mergeLabels(@NonNull
                                                        @NonNull INDArray[][] labelsToMerge,
                                                        INDArray[][] labelMasksToMerge,
                                                        int inOutIdx)

Extract out the specified column, and merge the specified label and label mask arrays (i.e., concatenate the examples)

Parameters:

labelsToMerge - Features to merge. Will use featuresToMerge[all][inOutIdx]

labelMasksToMerge - Mask arrays to merge. May be null

Returns:

Merged features and mask. Mask may be null

merge2d

public static Pair<INDArray,INDArray> merge2d(@NonNull
                                                    @NonNull INDArray[][] arrays,
                                                    INDArray[][] masks,
                                                    int inOutIdx)

Merge the specified 2d arrays and masks. See mergeFeatures(INDArray[], INDArray[]) and mergeLabels(INDArray[], INDArray[])

Parameters:

arrays - Arrays to merge

masks - Mask arrays to merge

inOutIdx - Index to extract out before merging

Returns:

Merged arrays and mask

merge2d

public static Pair<INDArray,INDArray> merge2d(INDArray[] arrays,
                                                    INDArray[] masks)

Merge the specified 2d arrays and masks. See mergeFeatures(INDArray[], INDArray[]) and mergeLabels(INDArray[], INDArray[])

Parameters:

arrays - Arrays to merge

masks - Mask arrays to merge

Returns:

Merged arrays and mask

mergePerOutputMasks2d

public static INDArray mergePerOutputMasks2d(long[] outShape,
                                             INDArray[][] arrays,
                                             INDArray[][] masks,
                                             int inOutIdx)

mergePerOutputMasks2d

@Deprecated
public static INDArray mergePerOutputMasks2d(long[] outShape,
                                             INDArray[] arrays,
                                             INDArray[] masks)

Deprecated.

USe mergeMasks2d(long[], INDArray[], INDArray[])

mergeMasks2d

public static INDArray mergeMasks2d(long[] outShape,
                                    INDArray[] arrays,
                                    INDArray[] masks)

mergeMasks4d

public static INDArray mergeMasks4d(INDArray[] featuresOrLabels,
                                    INDArray[] masks)

mergeTimeSeries

public static Pair<INDArray,INDArray> mergeTimeSeries(INDArray[][] arrays,
                                                            INDArray[][] masks,
                                                            int inOutIdx)

Merge the specified time series (3d) arrays and masks. See mergeFeatures(INDArray[], INDArray[]) and mergeLabels(INDArray[], INDArray[])

Parameters:

arrays - Arrays to merge

masks - Mask arrays to merge

inOutIdx - Index to extract out before merging

Returns:

Merged arrays and mask

mergeTimeSeries

public static Pair<INDArray,INDArray> mergeTimeSeries(INDArray[] arrays,
                                                            INDArray[] masks)

Merge the specified time series (3d) arrays and masks. See mergeFeatures(INDArray[], INDArray[]) and mergeLabels(INDArray[], INDArray[])

Parameters:

arrays - Arrays to merge

masks - Mask arrays to merge

Returns:

Merged arrays and mask

merge4d

public static Pair<INDArray,INDArray> merge4d(INDArray[][] arrays,
                                                    INDArray[][] masks,
                                                    int inOutIdx)

Merge the specified 4d arrays and masks. See mergeFeatures(INDArray[], INDArray[]) and mergeLabels(INDArray[], INDArray[])

Parameters:

arrays - Arrays to merge

masks - Mask arrays to merge

inOutIdx - Index to extract out before merging

Returns:

Merged arrays and mask

merge4d

public static Pair<INDArray,INDArray> merge4d(INDArray[] arrays,
                                                    INDArray[] masks)

Merge the specified 4d arrays and masks. See mergeFeatures(INDArray[], INDArray[]) and mergeLabels(INDArray[], INDArray[])

Parameters:

arrays - Arrays to merge

masks - Mask arrays to merge

Returns:

Merged arrays and mask