org.datavec.api.transform

Class TransformProcess.Builder

java.lang.Object

org.datavec.api.transform.TransformProcess.Builder

Enclosing class:

TransformProcess

public static class TransformProcess.Builder
extends Object

Builder class for constructing a TransformProcess

Constructor Summary

Constructors

Constructor and Description
Builder(Schema initialSchema)

Method Summary

Modifier and Type	Method and Description
TransformProcess.Builder	appendStringColumnTransform(String column, String toAppend) Append a String to a specified column
TransformProcess	build() Create the TransformProcess object
TransformProcess.Builder	calculateSortedRank(String newColumnName, String sortOnColumn, WritableComparator comparator) CalculateSortedRank: calculate the rank of each example, after sorting example.
TransformProcess.Builder	calculateSortedRank(String newColumnName, String sortOnColumn, WritableComparator comparator, boolean ascending) CalculateSortedRank: calculate the rank of each example, after sorting example.
TransformProcess.Builder	categoricalToInteger(String... columnNames) Convert the specified column(s) from a categorical representation to an integer representation.
TransformProcess.Builder	categoricalToOneHot(String... columnNames) Convert the specified column(s) from a categorical representation to a one-hot representation.
TransformProcess.Builder	conditionalCopyValueTransform(String columnToReplace, String sourceColumn, Condition condition) Replace the value in a specified column with a new value taken from another column, if a condition is satisfied/true. Note that the condition can be any generic condition, including on other column(s), different to the column that will be modified if the condition is satisfied/true.
TransformProcess.Builder	conditionalReplaceValueTransform(String column, Writable newValue, Condition condition) Replace the values in a specified column with a specified new value, if some condition holds.
TransformProcess.Builder	convertFromSequence() Convert a sequence to a set of individual values (by treating each value in each sequence as a separate example)
TransformProcess.Builder	convertToSequence(String keyColumn, SequenceComparator comparator) Convert a set of independent records/examples into a sequence, according to some key.
TransformProcess.Builder	doubleColumnsMathOp(String newColumnName, MathOp mathOp, String... columnNames) Calculate and add a new double column by performing a mathematical operation on a number of existing columns.
TransformProcess.Builder	doubleMathOp(String columnName, MathOp mathOp, double scalar) Perform a mathematical operation (add, subtract, scalar max etc) on the specified double column, with a scalar
TransformProcess.Builder	duplicateColumn(String column, String newName) Duplicate a single column
TransformProcess.Builder	duplicateColumns(List<String> columnNames, List<String> newNames) Duplicate a set of columns
TransformProcess.Builder	filter(Filter filter) Add a filter operation to be executed after the previously-added operations have been executed
TransformProcess.Builder	integerColumnsMathOp(String newColumnName, MathOp mathOp, String... columnNames) Calculate and add a new integer column by performing a mathematical operation on a number of existing columns.
TransformProcess.Builder	integerMathOp(String column, MathOp mathOp, int scalar) Perform a mathematical operation (add, subtract, scalar max etc) on the specified integer column, with a scalar
TransformProcess.Builder	integerToCategorical(String columnName, List<String> categoryStateNames) Convert the specified column from an integer representation (assume values 0 to numCategories-1) to a categorical representation, given the specified state names
TransformProcess.Builder	integerToCategorical(String columnName, Map<Integer,String> categoryIndexNameMap) Convert the specified column from an integer representation to a categorical representation, given the specified mapping between integer indexes and state names
TransformProcess.Builder	longColumnsMathOp(String newColumnName, MathOp mathOp, String... columnNames) Calculate and add a new long column by performing a mathematical operation on a number of existing columns.
TransformProcess.Builder	longMathOp(String columnName, MathOp mathOp, long scalar) Perform a mathematical operation (add, subtract, scalar max etc) on the specified long column, with a scalar
TransformProcess.Builder	normalize(String column, Normalize type, DataAnalysis da) Normalize the specified column with a given type of normalization
TransformProcess.Builder	reduce(IReducer reducer) Reduce (i.e., aggregate/combine) a set of examples (typically by key).
TransformProcess.Builder	reduceSequenceByWindow(IReducer reducer, WindowFunction windowFunction) Reduce (i.e., aggregate/combine) a set of sequence examples - for each sequence individually - using a window function.
TransformProcess.Builder	removeAllColumnsExceptFor(Collection<String> columnNames) Remove all columns, except for those that are specified here
TransformProcess.Builder	removeAllColumnsExceptFor(String... columnNames) Remove all columns, except for those that are specified here
TransformProcess.Builder	removeColumns(Collection<String> columnNames) Remove all of the specified columns, by name
TransformProcess.Builder	removeColumns(String... columnNames) Remove all of the specified columns, by name
TransformProcess.Builder	renameColumn(String oldName, String newName) Rename a single column
TransformProcess.Builder	renameColumns(List<String> oldNames, List<String> newNames) Rename multiple columns
TransformProcess.Builder	reorderColumns(String... newOrder) Reorder the columns using a partial or complete new ordering.
TransformProcess.Builder	splitSequence(SequenceSplit split) Split sequences into 1 or more other sequences.
TransformProcess.Builder	stringMapTransform(String columnName, Map<String,String> mapping) Replace one or more String values in the specified column with new values.
TransformProcess.Builder	stringRemoveWhitespaceTransform(String columnName) Remove all whitespace characters from the values in the specified String column
TransformProcess.Builder	stringToCategorical(String columnName, List<String> stateNames) Convert the specified String column to a categorical column.
TransformProcess.Builder	stringToTimeTransform(String column, String format, org.joda.time.DateTimeZone dateTimeZone) Convert a String column (containing a date/time String) to a time column (by parsing the date/time String)
TransformProcess.Builder	timeMathOp(String columnName, MathOp mathOp, long timeQuantity, TimeUnit timeUnit) Perform a mathematical operation (add, subtract, scalar min/max only) on the specified time column
TransformProcess.Builder	transform(Transform transform) Add a transformation to be executed after the previously-added operations have been executed

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Builder

public Builder(Schema initialSchema)

Method Detail

transform

public TransformProcess.Builder transform(Transform transform)

Add a transformation to be executed after the previously-added operations have been executed

Parameters:

transform - Transform to execute

filter

public TransformProcess.Builder filter(Filter filter)

Add a filter operation to be executed after the previously-added operations have been executed

Parameters:

filter - Filter operation to execute

removeColumns

public TransformProcess.Builder removeColumns(String... columnNames)

Remove all of the specified columns, by name

Parameters:

columnNames - Names of the columns to remove

removeColumns

public TransformProcess.Builder removeColumns(Collection<String> columnNames)

Remove all of the specified columns, by name

Parameters:

columnNames - Names of the columns to remove

removeAllColumnsExceptFor

public TransformProcess.Builder removeAllColumnsExceptFor(String... columnNames)

Remove all columns, except for those that are specified here

Parameters:

columnNames - Names of the columns to keep

removeAllColumnsExceptFor

public TransformProcess.Builder removeAllColumnsExceptFor(Collection<String> columnNames)

Remove all columns, except for those that are specified here

Parameters:

columnNames - Names of the columns to keep

renameColumn

public TransformProcess.Builder renameColumn(String oldName,
                                             String newName)

Rename a single column

Parameters:

oldName - Original column name

newName - New column name

renameColumns

public TransformProcess.Builder renameColumns(List<String> oldNames,
                                              List<String> newNames)

Rename multiple columns

Parameters:

oldNames - List of original column names

newNames - List of new column names

reorderColumns

public TransformProcess.Builder reorderColumns(String... newOrder)

Reorder the columns using a partial or complete new ordering. If only some of the column names are specified for the new order, the remaining columns will be placed at the end, according to their current relative ordering

Parameters:

newOrder - Names of the columns, in the order they will appear in the output

duplicateColumn

public TransformProcess.Builder duplicateColumn(String column,
                                                String newName)

Duplicate a single column

Parameters:

column - Name of the column to duplicate

newName - Name of the new (duplicate) column

duplicateColumns

public TransformProcess.Builder duplicateColumns(List<String> columnNames,
                                                 List<String> newNames)

Duplicate a set of columns

Parameters:

columnNames - Names of the columns to duplicate

newNames - Names of the new (duplicated) columns

integerMathOp

public TransformProcess.Builder integerMathOp(String column,
                                              MathOp mathOp,
                                              int scalar)

Perform a mathematical operation (add, subtract, scalar max etc) on the specified integer column, with a scalar

Parameters:

column - The integer column to perform the operation on

mathOp - The mathematical operation

scalar - The scalar value to use in the mathematical operation

integerColumnsMathOp

public TransformProcess.Builder integerColumnsMathOp(String newColumnName,
                                                     MathOp mathOp,
                                                     String... columnNames)

Calculate and add a new integer column by performing a mathematical operation on a number of existing columns. New column is added to the end.

Parameters:

newColumnName - Name of the new/derived column

mathOp - Mathematical operation to execute on the columns

columnNames - Names of the columns to use in the mathematical operation

longMathOp

public TransformProcess.Builder longMathOp(String columnName,
                                           MathOp mathOp,
                                           long scalar)

Perform a mathematical operation (add, subtract, scalar max etc) on the specified long column, with a scalar

Parameters:

columnName - The long column to perform the operation on

mathOp - The mathematical operation

scalar - The scalar value to use in the mathematical operation

longColumnsMathOp

public TransformProcess.Builder longColumnsMathOp(String newColumnName,
                                                  MathOp mathOp,
                                                  String... columnNames)

Calculate and add a new long column by performing a mathematical operation on a number of existing columns. New column is added to the end.

Parameters:

newColumnName - Name of the new/derived column

mathOp - Mathematical operation to execute on the columns

columnNames - Names of the columns to use in the mathematical operation

doubleMathOp

public TransformProcess.Builder doubleMathOp(String columnName,
                                             MathOp mathOp,
                                             double scalar)

Perform a mathematical operation (add, subtract, scalar max etc) on the specified double column, with a scalar

Parameters:

columnName - The double column to perform the operation on

mathOp - The mathematical operation

scalar - The scalar value to use in the mathematical operation

doubleColumnsMathOp

public TransformProcess.Builder doubleColumnsMathOp(String newColumnName,
                                                    MathOp mathOp,
                                                    String... columnNames)

Calculate and add a new double column by performing a mathematical operation on a number of existing columns. New column is added to the end.

Parameters:

newColumnName - Name of the new/derived column

mathOp - Mathematical operation to execute on the columns

columnNames - Names of the columns to use in the mathematical operation

timeMathOp

public TransformProcess.Builder timeMathOp(String columnName,
                                           MathOp mathOp,
                                           long timeQuantity,
                                           TimeUnit timeUnit)

Perform a mathematical operation (add, subtract, scalar min/max only) on the specified time column

Parameters:

columnName - The integer column to perform the operation on

mathOp - The mathematical operation

timeQuantity - The quantity used in the mathematical op

timeUnit - The unit that timeQuantity is specified in

categoricalToOneHot

public TransformProcess.Builder categoricalToOneHot(String... columnNames)

Convert the specified column(s) from a categorical representation to a one-hot representation. This involves the creation of multiple new columns each.

Parameters:

columnNames - Names of the categorical column(s) to convert to a one-hot representation

categoricalToInteger

public TransformProcess.Builder categoricalToInteger(String... columnNames)

Convert the specified column(s) from a categorical representation to an integer representation. This will replace the specified categorical column(s) with an integer repreesentation, where each integer has the value 0 to numCategories-1.

Parameters:

columnNames - Name of the categorical column(s) to convert to an integer representation

integerToCategorical

public TransformProcess.Builder integerToCategorical(String columnName,
                                                     List<String> categoryStateNames)

Convert the specified column from an integer representation (assume values 0 to numCategories-1) to a categorical representation, given the specified state names

Parameters:

columnName - Name of the column to convert

categoryStateNames - Names of the states for the categorical column

integerToCategorical

public TransformProcess.Builder integerToCategorical(String columnName,
                                                     Map<Integer,String> categoryIndexNameMap)

Convert the specified column from an integer representation to a categorical representation, given the specified mapping between integer indexes and state names

Parameters:

columnName - Name of the column to convert

categoryIndexNameMap - Names of the states for the categorical column

normalize

public TransformProcess.Builder normalize(String column,
                                          Normalize type,
                                          DataAnalysis da)

Normalize the specified column with a given type of normalization

Parameters:

column - Column to normalize

type - Type of normalization to apply

da - DataAnalysis object

convertToSequence

public TransformProcess.Builder convertToSequence(String keyColumn,
                                                  SequenceComparator comparator)

Convert a set of independent records/examples into a sequence, according to some key. Within each sequence, values are ordered using the provided SequenceComparator

Parameters:

keyColumn - Column to use as a key (values with the same key will be combined into sequences)

comparator - A SequenceComparator to order the values within each sequence (for example, by time or String order)

convertFromSequence

public TransformProcess.Builder convertFromSequence()

Convert a sequence to a set of individual values (by treating each value in each sequence as a separate example)

splitSequence

public TransformProcess.Builder splitSequence(SequenceSplit split)

Split sequences into 1 or more other sequences. Used for example to split large sequences into a set of smaller sequences

Parameters:

split - SequenceSplit that defines how splits will occur

reduce

public TransformProcess.Builder reduce(IReducer reducer)

Reduce (i.e., aggregate/combine) a set of examples (typically by key). Note: In the current implementation, reduction operations can be performed only on standard (i.e., non-sequence) data

Parameters:

reducer - Reducer to use

reduceSequenceByWindow

public TransformProcess.Builder reduceSequenceByWindow(IReducer reducer,
                                                       WindowFunction windowFunction)

Reduce (i.e., aggregate/combine) a set of sequence examples - for each sequence individually - using a window function. For example, take all records/examples in each 24-hour period (i.e., using window function), and convert them into a singe value (using the reducer). In this example, the output is a sequence, with time period of 24 hours.

Parameters:

reducer - Reducer to use to reduce each window

windowFunction - Window function to find apply on each sequence individually

calculateSortedRank

public TransformProcess.Builder calculateSortedRank(String newColumnName,
                                                    String sortOnColumn,
                                                    WritableComparator comparator)

CalculateSortedRank: calculate the rank of each example, after sorting example. For example, we might have some numerical "score" column, and we want to know for the rank (sort order) for each example, according to that column.
The rank of each example (after sorting) will be added in a new Long column. Indexing is done from 0; examples will have values 0 to dataSetSize-1.

Currently, CalculateSortedRank can only be applied on standard (i.e., non-sequence) data Furthermore, the current implementation can only sort on one column

Parameters:

newColumnName - Name of the new column (will contain the rank for each example)

sortOnColumn - Column to sort on

comparator - Comparator used to sort examples

calculateSortedRank

public TransformProcess.Builder calculateSortedRank(String newColumnName,
                                                    String sortOnColumn,
                                                    WritableComparator comparator,
                                                    boolean ascending)

CalculateSortedRank: calculate the rank of each example, after sorting example. For example, we might have some numerical "score" column, and we want to know for the rank (sort order) for each example, according to that column.
The rank of each example (after sorting) will be added in a new Long column. Indexing is done from 0; examples will have values 0 to dataSetSize-1.

Currently, CalculateSortedRank can only be applied on standard (i.e., non-sequence) data Furthermore, the current implementation can only sort on one column

Parameters:

newColumnName - Name of the new column (will contain the rank for each example)

sortOnColumn - Column to sort on

comparator - Comparator used to sort examples

ascending - If true: sort ascending. False: descending

stringToCategorical

public TransformProcess.Builder stringToCategorical(String columnName,
                                                    List<String> stateNames)

Convert the specified String column to a categorical column. The state names must be provided.

Parameters:

columnName - Name of the String column to convert to categorical

stateNames - State names of the category

stringRemoveWhitespaceTransform

public TransformProcess.Builder stringRemoveWhitespaceTransform(String columnName)

Remove all whitespace characters from the values in the specified String column

Parameters:

columnName - Name of the column to remove whitespace from

stringMapTransform

public TransformProcess.Builder stringMapTransform(String columnName,
                                                   Map<String,String> mapping)

Replace one or more String values in the specified column with new values.

Keys in the map are the original values; the Values in the map are their replacements. If a String appears in the data but does not appear in the provided map (as a key), that String values will not be modified.

Parameters:

columnName - Name of the column in which to do replacement

mapping - Map of oldValues -> newValues

stringToTimeTransform

public TransformProcess.Builder stringToTimeTransform(String column,
                                                      String format,
                                                      org.joda.time.DateTimeZone dateTimeZone)

Convert a String column (containing a date/time String) to a time column (by parsing the date/time String)

Parameters:

column - String column containing the date/time Strings

format - Format of the strings. Time format is specified as per http://www.joda.org/joda-time/apidocs/org/joda/time/format/DateTimeFormat.html

dateTimeZone - Timezone of the column

appendStringColumnTransform

public TransformProcess.Builder appendStringColumnTransform(String column,
                                                            String toAppend)

Append a String to a specified column

Parameters:

column - Column to append the value to

toAppend - String to append to the end of each writable

conditionalReplaceValueTransform

public TransformProcess.Builder conditionalReplaceValueTransform(String column,
                                                                 Writable newValue,
                                                                 Condition condition)

Replace the values in a specified column with a specified new value, if some condition holds. If the condition does not hold, the original values are not modified.

Parameters:

column - Column to operate on

newValue - Value to use as replacement, if condition is satisfied

condition - Condition that must be satisfied for replacement

conditionalCopyValueTransform

public TransformProcess.Builder conditionalCopyValueTransform(String columnToReplace,
                                                              String sourceColumn,
                                                              Condition condition)

Replace the value in a specified column with a new value taken from another column, if a condition is satisfied/true.
Note that the condition can be any generic condition, including on other column(s), different to the column that will be modified if the condition is satisfied/true.

Parameters:

columnToReplace - Name of the column in which values will be replaced (if condition is satisfied)

sourceColumn - Name of the column from which the new values will be

condition - Condition to use

build

public TransformProcess build()

Create the TransformProcess object