com.salesforce.op.stages.impl.feature
OPCollectionTransformer
Related Doc:
package feature
Function that transforms the relevant entry of the collection based on the supplied transformer.
Function that transforms the relevant entry of the collection based on the supplied transformer. OPLists and OPSets have their entries transformed, and OPMaps have just their values transformed.
unique name of the operation this stage performs
unique name of the operation this stage performs
Function that takes the values contained in our FeatureTypes and converts them to/from types that the supplied UnaryTransformer operates on and applies the unary transformation.
Function that takes the values contained in our FeatureTypes and converts them to/from types that the supplied UnaryTransformer operates on and applies the unary transformation. Note that this does not do any checks on the output, so if the unary transformer function produces a null, then we leave that in the map (Map[String, Long] can still contain eg. ("a" -> null))
input value contained in type ICol (eg. if ICol is TextMap, then value is Map[String, String])
transformed output value contained
UnaryTransformer on non-collection types that we wish to use on collection types
type tag for input
type tag for input
type tag for output
type tag for output
type tag for output value
type tag for output value
uid for instance
uid for instance
Abstract base class for a set of transformer wrappers that allow unary transformers between non-collection types to be used on collection types. For example, we can use a UnaryLambdaTransformer[Email, Integer] on a map's values, creating a UnaryLambdaTransformer[EmailMap, IntegralMap]. This base class will be inherited by concrete classes for OPMaps, OPList, and OPSets (in order to enforce not allowing these collection types to be transformed into each other, eg. no MultiPickList to RealMap transformations).
The OP type hierarchy does not allow direct type checking of such transformer wrappers (eg. Real#Value is Option[Double] and RealMap#Value is Map[String, Double], so there's no way to enforce that a RealMap can only hold what is contained in a Real) since the types themselves are not created with typetags for performance reasons. However, we can still enforce that operations like building a UnaryLambdaTransformer[RealMap, StringMap] from a UnaryLambdaTransformer[Real, Integer] is not possible by using the Spark types in validateTypes.
input feature type for supplied non-collection transformer
output feature type for supplied non-collection transformer
input feature type for desired collection transformer
output feature type for desired collection transformer