Class

com.salesforce.op.stages.sparkwrappers.specific

OpTransformerWrapper

Related Doc: package specific

Permalink

class OpTransformerWrapper[I <: FeatureType, O <: FeatureType, T <: InOutTransformer] extends SwUnaryTransformer[I, O, T]

Wraps a spark ML transformer with setable input and output columns. Those transformers that fall in this case, include those that inherit from org.apache.spark.ml.UnaryEstimator, as well as others such as OneHotEncoder, org.apache.spark.ml.feature.Binarizer, org.apache.spark.ml.feature.VectorSlicer, org.apache.spark.ml.feature.HashingTF, org.apache.spark.ml.feature.StopWordsRemover, org.apache.spark.ml.feature.IndexToString, org.apache.spark.ml.feature.StringIndexer. Their defining characteristic is that they take one column as input, and output one column as result.

I

The type of the input feature

O

The type of the output feature (result of transformation)

T

type of spark transformer to wrap

Linear Supertypes
SwUnaryTransformer[I, O, T], SwTransformer1[I, O, T], SparkWrapperParams[T], OpPipelineStage1[I, O], HasIn1, OpPipelineStage[O], OpPipelineStageBase, MLWritable, OpPipelineStageParams, InputParams, Transformer, PipelineStage, Logging, Params, Serializable, Serializable, Identifiable, AnyRef, Any
Known Subclasses
OpIndexToString
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. OpTransformerWrapper
  2. SwUnaryTransformer
  3. SwTransformer1
  4. SparkWrapperParams
  5. OpPipelineStage1
  6. HasIn1
  7. OpPipelineStage
  8. OpPipelineStageBase
  9. MLWritable
  10. OpPipelineStageParams
  11. InputParams
  12. Transformer
  13. PipelineStage
  14. Logging
  15. Params
  16. Serializable
  17. Serializable
  18. Identifiable
  19. AnyRef
  20. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new OpTransformerWrapper(transformer: T, uid: String = UID[OpTransformerWrapper[I, O, T]])(implicit tti: scala.reflect.api.JavaUniverse.TypeTag[I], tto: scala.reflect.api.JavaUniverse.TypeTag[O], ttov: scala.reflect.api.JavaUniverse.TypeTag[O.Value])

    Permalink

    transformer

    The spark ML transformer that's being wrapped

    uid

    stage uid

Type Members

  1. final type InputFeatures = FeatureLike[I]

    Permalink
    Definition Classes
    OpPipelineStage1 → OpPipelineStage → InputParams

  2. final type OutputFeatures = FeatureLike[O]

    Permalink
    Definition Classes
    OpPipelineStage → OpPipelineStageBase

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def $[T](param: Param[T]): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  4. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  5. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  6. final def checkInputLength(features: Array[_]): Boolean

    Permalink
    Definition Classes
    OpPipelineStage1 → InputParams

  7. def checkSerializable: Try[Unit]

    Permalink
    Definition Classes
    OpPipelineStageBase

  8. final def clear(param: Param[_]): OpTransformerWrapper.this.type

    Permalink
    Definition Classes
    Params

  9. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  10. final def copy(extra: ParamMap): OpTransformerWrapper.this.type

    Permalink
    Definition Classes
    OpPipelineStageBase → Params

  11. def copyValues[T <: Params](to: T, extra: ParamMap): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  12. final def defaultCopy[T <: Params](extra: ParamMap): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  13. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  14. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  15. def explainParam(param: Param[_]): String

    Permalink
    Definition Classes
    Params

  16. def explainParams(): String

    Permalink
    Definition Classes
    Params

  17. final def extractParamMap(): ParamMap

    Permalink
    Definition Classes
    Params

  18. final def extractParamMap(extra: ParamMap): ParamMap

    Permalink
    Definition Classes
    Params

  19. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  20. final def get[T](param: Param[T]): Option[T]

    Permalink
    Definition Classes
    Params

  21. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  22. final def getDefault[T](param: Param[T]): Option[T]

    Permalink
    Definition Classes
    Params

  23. final def getInputFeature[T <: FeatureType](i: Int): Option[FeatureLike[T]]

    Permalink
    Definition Classes
    InputParams

  24. final def getInputFeatures(): Array[OPFeature]

    Permalink
    Definition Classes
    InputParams

  25. final def getInputSchema(): StructType

    Permalink
    Definition Classes
    OpPipelineStageParams

  26. final def getMetadata(): Metadata

    Permalink
    Definition Classes
    OpPipelineStageParams

  27. final def getOrDefault[T](param: Param[T]): T

    Permalink
    Definition Classes
    Params

  28. def getOutput(): FeatureLike[O]

    Permalink
    Definition Classes
    OpPipelineStage1 → OpPipelineStageBase

  29. final def getOutputFeatureName: String

    Permalink
    Definition Classes
    OpPipelineStage

  30. def getParam(paramName: String): Param[Any]

    Permalink
    Definition Classes
    Params

  31. def getSparkMlStage(): Option[T]

    Permalink
    Definition Classes
    SparkWrapperParams

  32. def getStageSavePath(): Option[String]

    Permalink
    Definition Classes
    SparkWrapperParams

  33. final def getTransientFeature(i: Int): Option[TransientFeature]

    Permalink
    Definition Classes
    InputParams

  34. final def getTransientFeatures(): Array[TransientFeature]

    Permalink
    Definition Classes
    InputParams

  35. final def hasDefault[T](param: Param[T]): Boolean

    Permalink
    Definition Classes
    Params

  36. def hasParam(paramName: String): Boolean

    Permalink
    Definition Classes
    Params

  37. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  38. final def in1: TransientFeature

    Permalink
    Attributes
    protected
    Definition Classes
    HasIn1

  39. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  40. final def inputAsArray(in: InputFeatures): Array[OPFeature]

    Permalink
    Definition Classes
    OpPipelineStage1 → InputParams

  41. val inputParamName: String

    Permalink

    name of spark parameter that sets the first input column

    name of spark parameter that sets the first input column

    Definition Classes
    SwUnaryTransformer → SwTransformer1

  42. final def isDefined(param: Param[_]): Boolean

    Permalink
    Definition Classes
    Params

  43. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  44. final def isSet(param: Param[_]): Boolean

    Permalink
    Definition Classes
    Params

  45. def isTraceEnabled(): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  46. def log: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  47. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  48. def logDebug(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  49. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  50. def logError(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  51. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  52. def logInfo(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  53. def logName: String

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  54. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  55. def logTrace(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  56. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  57. def logWarning(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  58. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  59. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  60. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  61. def onGetMetadata(): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    OpPipelineStageParams

  62. def onSetInput(): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    OpPipelineStageBase

  63. val operationName: String

    Permalink

    unique name of the operation this stage performs

    unique name of the operation this stage performs

    Definition Classes
    SwUnaryTransformer → OpPipelineStageBase

  64. final def outputAsArray(out: OutputFeatures): Array[OPFeature]

    Permalink
    Definition Classes
    OpPipelineStage → OpPipelineStageBase

  65. def outputFeatureUid: String

    Permalink
    Attributes
    protected[com.salesforce.op]
    Definition Classes
    OpPipelineStage1 → OpPipelineStage

  66. def outputIsResponse: Boolean

    Permalink
    Definition Classes
    OpPipelineStage

  67. val outputParamName: String

    Permalink

    name of spark parameter that sets the first output column

    name of spark parameter that sets the first output column

    Definition Classes
    SwUnaryTransformer → SwTransformer1

  68. lazy val params: Array[Param[_]]

    Permalink
    Definition Classes
    Params

  69. def save(path: String): Unit

    Permalink
    Definition Classes
    MLWritable
    Annotations
    @Since( "1.6.0" ) @throws( ... )

  70. final def set(paramPair: ParamPair[_]): OpTransformerWrapper.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  71. final def set(param: String, value: Any): OpTransformerWrapper.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  72. final def set[T](param: Param[T], value: T): OpTransformerWrapper.this.type

    Permalink
    Definition Classes
    Params

  73. final def setDefault(paramPairs: ParamPair[_]*): OpTransformerWrapper.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  74. final def setDefault[T](param: Param[T], value: T): OpTransformerWrapper.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  75. final def setInput(features: InputFeatures): OpTransformerWrapper.this.type

    Permalink
    Definition Classes
    OpPipelineStageBase

  76. final def setInputFeatures[S <: OPFeature](features: Array[S]): OpTransformerWrapper.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    InputParams

  77. final def setMetadata(m: Metadata): OpTransformerWrapper.this.type

    Permalink
    Definition Classes
    OpPipelineStageParams

  78. def setOutputFeatureName(name: String): OpTransformerWrapper.this.type

    Permalink
    Definition Classes
    OpPipelineStage

  79. def setSparkMlStage(stage: Option[T]): OpTransformerWrapper.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    SparkWrapperParams

  80. def setStageSavePath(path: String): OpTransformerWrapper.this.type

    Permalink
    Definition Classes
    SparkWrapperParams

  81. final val sparkInputColParamNames: StringArrayParam

    Permalink
    Definition Classes
    SparkWrapperParams

  82. final val sparkMlStage: SparkStageParam[T]

    Permalink
    Definition Classes
    SparkWrapperParams

  83. final val sparkOutputColParamNames: StringArrayParam

    Permalink
    Definition Classes
    SparkWrapperParams

  84. final def stageName: String

    Permalink
    Definition Classes
    OpPipelineStageBase

  85. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  86. def toString(): String

    Permalink
    Definition Classes
    Identifiable → AnyRef → Any

  87. def transform(dataset: Dataset[_]): DataFrame

    Permalink
    Definition Classes
    SwTransformer1 → Transformer

  88. def transform(dataset: Dataset[_], paramMap: ParamMap): DataFrame

    Permalink
    Definition Classes
    Transformer
    Annotations
    @Since( "2.0.0" )

  89. def transform(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): DataFrame

    Permalink
    Definition Classes
    Transformer
    Annotations
    @Since( "2.0.0" ) @varargs()

  90. final def transformSchema(schema: StructType): StructType

    Permalink
    Definition Classes
    OpPipelineStageBase

  91. def transformSchema(schema: StructType, logging: Boolean): StructType

    Permalink
    Attributes
    protected
    Definition Classes
    PipelineStage
    Annotations
    @DeveloperApi()

  92. val transformer: T

    Permalink

    The spark ML transformer that's being wrapped

  93. implicit val tti: scala.reflect.api.JavaUniverse.TypeTag[I]

    Permalink
    Definition Classes
    SwUnaryTransformer → SwTransformer1

  94. implicit val tto: scala.reflect.api.JavaUniverse.TypeTag[O]

    Permalink
    Definition Classes
    SwUnaryTransformer → OpPipelineStage1

  95. implicit val ttov: scala.reflect.api.JavaUniverse.TypeTag[O.Value]

    Permalink
    Definition Classes
    SwUnaryTransformer → OpPipelineStage1

  96. val uid: String

    Permalink

    stage uid

    stage uid

    Definition Classes
    SwUnaryTransformer → Identifiable

  97. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  98. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  99. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  100. final def write: MLWriter

    Permalink
    Definition Classes
    OpPipelineStageBase → MLWritable

Inherited from SwUnaryTransformer[I, O, T]

Inherited from SwTransformer1[I, O, T]

Inherited from SparkWrapperParams[T]

Inherited from OpPipelineStage1[I, O]

Inherited from HasIn1

Inherited from OpPipelineStage[O]

Inherited from OpPipelineStageBase

Inherited from MLWritable

Inherited from OpPipelineStageParams

Inherited from InputParams

Inherited from Transformer

Inherited from PipelineStage

Inherited from Logging

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

Ungrouped