Class/Object

com.salesforce.op.stages.impl.preparators

SanityChecker

Related Docs: object SanityChecker | package preparators

Permalink

class SanityChecker extends BinaryEstimator[RealNN, OPVector, OPVector] with SanityCheckerParams with AllowLabelAsInput[OPVector]

The SanityChecker checks for potential problems with computed features in a supervised learning setting.

There is an Estimator step, which outputs statistics on the incoming data, as well as the names of features which should be dropped from the feature vector. The transformer step applies the action of actually removing the offending features from the feature vector.

Linear Supertypes
AllowLabelAsInput[OPVector], SanityCheckerParams, BinaryEstimator[RealNN, OPVector, OPVector], OpPipelineStage2[RealNN, OPVector, OPVector], HasIn2, HasIn1, OpPipelineStage[OPVector], OpPipelineStageBase, MLWritable, OpPipelineStageParams, InputParams, Estimator[BinaryModel[RealNN, OPVector, OPVector]], PipelineStage, Logging, Params, Serializable, Serializable, Identifiable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SanityChecker
  2. AllowLabelAsInput
  3. SanityCheckerParams
  4. BinaryEstimator
  5. OpPipelineStage2
  6. HasIn2
  7. HasIn1
  8. OpPipelineStage
  9. OpPipelineStageBase
  10. MLWritable
  11. OpPipelineStageParams
  12. InputParams
  13. Estimator
  14. PipelineStage
  15. Logging
  16. Params
  17. Serializable
  18. Serializable
  19. Identifiable
  20. AnyRef
  21. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new SanityChecker(uid: String = UID[SanityChecker])

    Permalink

Type Members

  1. final type InputFeatures = (FeatureLike[RealNN], FeatureLike[OPVector])

    Permalink
    Definition Classes
    OpPipelineStage2 → OpPipelineStage → InputParams

  2. final type OutputFeatures = FeatureLike[OPVector]

    Permalink
    Definition Classes
    OpPipelineStage → OpPipelineStageBase

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def $[T](param: Param[T]): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  4. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  5. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  6. final val categoricalLabel: BooleanParam

    Permalink
    Definition Classes
    SanityCheckerParams

  7. final def checkInputLength(features: Array[_]): Boolean

    Permalink
    Definition Classes
    OpPipelineStage2 → InputParams

  8. final val checkSample: DoubleParam

    Permalink
    Definition Classes
    SanityCheckerParams

  9. final def checkSerializable: Try[Unit]

    Permalink
    Definition Classes
    BinaryEstimator → OpPipelineStageBase

  10. final def clear(param: Param[_]): SanityChecker.this.type

    Permalink
    Definition Classes
    Params

  11. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  12. val convertI1: FeatureTypeSparkConverter[RealNN]

    Permalink
    Definition Classes
    BinaryEstimator

  13. val convertI2: FeatureTypeSparkConverter[OPVector]

    Permalink
    Definition Classes
    BinaryEstimator

  14. final def copy(extra: ParamMap): SanityChecker.this.type

    Permalink
    Definition Classes
    OpPipelineStageBase → Params

  15. def copyValues[T <: Params](to: T, extra: ParamMap): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  16. final val correlationExclusion: Param[String]

    Permalink
    Definition Classes
    SanityCheckerParams

  17. final val correlationType: Param[String]

    Permalink
    Definition Classes
    SanityCheckerParams

  18. final def defaultCopy[T <: Params](extra: ParamMap): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  19. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  20. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  21. def explainParam(param: Param[_]): String

    Permalink
    Definition Classes
    Params

  22. def explainParams(): String

    Permalink
    Definition Classes
    Params

  23. final def extractParamMap(): ParamMap

    Permalink
    Definition Classes
    Params

  24. final def extractParamMap(extra: ParamMap): ParamMap

    Permalink
    Definition Classes
    Params

  25. final val featureLabelCorrOnly: BooleanParam

    Permalink
    Definition Classes
    SanityCheckerParams

  26. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  27. def fit(dataset: Dataset[_]): BinaryModel[RealNN, OPVector, OPVector]

    Permalink
    Definition Classes
    BinaryEstimator → Estimator

  28. def fit(dataset: Dataset[_], paramMaps: Array[ParamMap]): Seq[BinaryModel[RealNN, OPVector, OPVector]]

    Permalink
    Definition Classes
    Estimator
    Annotations
    @Since( "2.0.0" )

  29. def fit(dataset: Dataset[_], paramMap: ParamMap): BinaryModel[RealNN, OPVector, OPVector]

    Permalink
    Definition Classes
    Estimator
    Annotations
    @Since( "2.0.0" )

  30. def fit(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): BinaryModel[RealNN, OPVector, OPVector]

    Permalink
    Definition Classes
    Estimator
    Annotations
    @Since( "2.0.0" ) @varargs()

  31. def fitFn(data: Dataset[(Option[Double], Vector)]): BinaryModel[RealNN, OPVector, OPVector]

    Permalink

    The SanityChecker's core 'estimator' function, computes statistics on the features and the list of features to be removed.

    The SanityChecker's core 'estimator' function, computes statistics on the features and the list of features to be removed.

    Definition Classes
    SanityChecker → BinaryEstimator

  32. def fraction(totalSize: Long): Double

    Permalink

    Estimate of a fraction of data that is being checked.

  33. final def get[T](param: Param[T]): Option[T]

    Permalink
    Definition Classes
    Params

  34. def getCategoricalLabel: Boolean

    Permalink
    Definition Classes
    SanityCheckerParams

  35. def getCheckSample: Double

    Permalink
    Definition Classes
    SanityCheckerParams

  36. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  37. def getCorrelationExclusion: CorrelationExclusion

    Permalink
    Definition Classes
    SanityCheckerParams

  38. def getCorrelationType: CorrelationType

    Permalink
    Definition Classes
    SanityCheckerParams

  39. final def getDefault[T](param: Param[T]): Option[T]

    Permalink
    Definition Classes
    Params

  40. def getFeatureLabelCorrOnly: Boolean

    Permalink
    Definition Classes
    SanityCheckerParams

  41. final def getInputFeature[T <: FeatureType](i: Int): Option[FeatureLike[T]]

    Permalink
    Definition Classes
    InputParams

  42. final def getInputFeatures(): Array[OPFeature]

    Permalink
    Definition Classes
    InputParams

  43. final def getInputSchema(): StructType

    Permalink
    Definition Classes
    OpPipelineStageParams

  44. def getMaxCorrelation: Double

    Permalink
    Definition Classes
    SanityCheckerParams

  45. def getMaxCramersV: Double

    Permalink
    Definition Classes
    SanityCheckerParams

  46. def getMaxRuleConfidence: Double

    Permalink
    Definition Classes
    SanityCheckerParams

  47. final def getMetadata(): Metadata

    Permalink
    Definition Classes
    OpPipelineStageParams

  48. def getMinCorrelation: Double

    Permalink
    Definition Classes
    SanityCheckerParams

  49. def getMinRequiredRuleSupport: Double

    Permalink
    Definition Classes
    SanityCheckerParams

  50. def getMinVariance: Double

    Permalink
    Definition Classes
    SanityCheckerParams

  51. final def getOrDefault[T](param: Param[T]): T

    Permalink
    Definition Classes
    Params

  52. def getOutput(): FeatureLike[OPVector]

    Permalink
    Definition Classes
    OpPipelineStage2 → OpPipelineStageBase

  53. final def getOutputFeatureName: String

    Permalink
    Definition Classes
    OpPipelineStage

  54. def getParam(paramName: String): Param[Any]

    Permalink
    Definition Classes
    Params

  55. def getProtectTextSharedHash: Boolean

    Permalink
    Definition Classes
    SanityCheckerParams

  56. def getRemoveBadFeatures: Boolean

    Permalink
    Definition Classes
    SanityCheckerParams

  57. def getRemoveFeatureGroup: Boolean

    Permalink
    Definition Classes
    SanityCheckerParams

  58. def getSampleLowerLimit: Int

    Permalink
    Definition Classes
    SanityCheckerParams

  59. def getSampleSeed: Long

    Permalink
    Definition Classes
    SanityCheckerParams

  60. def getSampleUpperLimit: Int

    Permalink
    Definition Classes
    SanityCheckerParams

  61. final def getTransientFeature(i: Int): Option[TransientFeature]

    Permalink
    Definition Classes
    InputParams

  62. final def getTransientFeatures(): Array[TransientFeature]

    Permalink
    Definition Classes
    InputParams

  63. final def hasDefault[T](param: Param[T]): Boolean

    Permalink
    Definition Classes
    Params

  64. def hasParam(paramName: String): Boolean

    Permalink
    Definition Classes
    Params

  65. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  66. implicit val i1Encoder: Encoder[features.types.RealNN.Value]

    Permalink
    Definition Classes
    BinaryEstimator

  67. implicit val i2Encoder: Encoder[Value]

    Permalink
    Definition Classes
    BinaryEstimator

  68. final def in1: TransientFeature

    Permalink
    Attributes
    protected
    Definition Classes
    HasIn1

  69. final def in2: TransientFeature

    Permalink
    Attributes
    protected
    Definition Classes
    HasIn2

  70. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  71. final def inputAsArray(in: InputFeatures): Array[OPFeature]

    Permalink
    Definition Classes
    OpPipelineStage2 → InputParams

  72. final def isDefined(param: Param[_]): Boolean

    Permalink
    Definition Classes
    Params

  73. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  74. final def isSet(param: Param[_]): Boolean

    Permalink
    Definition Classes
    Params

  75. def isTraceEnabled(): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  76. def log: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  77. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  78. def logDebug(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  79. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  80. def logError(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  81. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  82. def logInfo(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  83. final val logLevel: Param[String]

    Permalink
    Definition Classes
    SanityCheckerParams

  84. def logName: String

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  85. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  86. def logTrace(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  87. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  88. def logWarning(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  89. final val maxCorrelation: DoubleParam

    Permalink
    Definition Classes
    SanityCheckerParams

  90. final val maxCramersV: DoubleParam

    Permalink
    Definition Classes
    SanityCheckerParams

  91. final val maxRuleConfidence: DoubleParam

    Permalink
    Definition Classes
    SanityCheckerParams

  92. final val minCorrelation: DoubleParam

    Permalink
    Definition Classes
    SanityCheckerParams

  93. final val minRequiredRuleSupport: DoubleParam

    Permalink
    Definition Classes
    SanityCheckerParams

  94. final val minVariance: DoubleParam

    Permalink
    Definition Classes
    SanityCheckerParams

  95. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  96. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  97. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  98. def onGetMetadata(): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    OpPipelineStageParams

  99. def onSetInput(): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    SanityChecker → OpPipelineStageBase

  100. val operationName: String

    Permalink
    Definition Classes
    BinaryEstimator → OpPipelineStageBase

  101. final def outputAsArray(out: OutputFeatures): Array[OPFeature]

    Permalink
    Definition Classes
    OpPipelineStage → OpPipelineStageBase

  102. def outputFeatureUid: String

    Permalink
    Attributes
    protected[com.salesforce.op]
    Definition Classes
    OpPipelineStage2 → OpPipelineStage

  103. def outputIsResponse: Boolean

    Permalink
    Definition Classes
    AllowLabelAsInput → OpPipelineStage

  104. lazy val params: Array[Param[_]]

    Permalink
    Definition Classes
    Params

  105. final val protectTextSharedHash: BooleanParam

    Permalink
    Definition Classes
    SanityCheckerParams

  106. final val removeBadFeatures: BooleanParam

    Permalink
    Definition Classes
    SanityCheckerParams

  107. final val removeFeatureGroup: BooleanParam

    Permalink
    Definition Classes
    SanityCheckerParams

  108. final val sampleLowerLimit: IntParam

    Permalink
    Definition Classes
    SanityCheckerParams

  109. final val sampleSeed: LongParam

    Permalink
    Definition Classes
    SanityCheckerParams

  110. final val sampleUpperLimit: IntParam

    Permalink
    Definition Classes
    SanityCheckerParams

  111. def save(path: String): Unit

    Permalink
    Definition Classes
    MLWritable
    Annotations
    @Since( "1.6.0" ) @throws( ... )

  112. final def set(paramPair: ParamPair[_]): SanityChecker.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  113. final def set(param: String, value: Any): SanityChecker.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  114. final def set[T](param: Param[T], value: T): SanityChecker.this.type

    Permalink
    Definition Classes
    Params

  115. def setCategoricalLabel(value: Boolean): SanityChecker.this.type

    Permalink
    Definition Classes
    SanityCheckerParams

  116. def setCheckSample(value: Double): SanityChecker.this.type

    Permalink
    Definition Classes
    SanityCheckerParams

  117. def setCorrelationExclusion(v: CorrelationExclusion): SanityChecker.this.type

    Permalink
    Definition Classes
    SanityCheckerParams

  118. def setCorrelationType(value: CorrelationType): SanityChecker.this.type

    Permalink
    Definition Classes
    SanityCheckerParams

  119. final def setDefault(paramPairs: ParamPair[_]*): SanityChecker.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  120. final def setDefault[T](param: Param[T], value: T): SanityChecker.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  121. def setFeatureLabelCorrOnly(value: Boolean): SanityChecker.this.type

    Permalink
    Definition Classes
    SanityCheckerParams

  122. final def setInput(features: InputFeatures): SanityChecker.this.type

    Permalink
    Definition Classes
    OpPipelineStageBase

  123. final def setInputFeatures[S <: OPFeature](features: Array[S]): SanityChecker.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    InputParams

  124. def setMaxCorrelation(value: Double): SanityChecker.this.type

    Permalink
    Definition Classes
    SanityCheckerParams

  125. def setMaxCramersV(value: Double): SanityChecker.this.type

    Permalink
    Definition Classes
    SanityCheckerParams

  126. def setMaxRuleConfidence(value: Double): SanityChecker.this.type

    Permalink
    Definition Classes
    SanityCheckerParams

  127. final def setMetadata(m: Metadata): SanityChecker.this.type

    Permalink
    Definition Classes
    OpPipelineStageParams

  128. def setMinCorrelation(value: Double): SanityChecker.this.type

    Permalink
    Definition Classes
    SanityCheckerParams

  129. def setMinRequiredRuleSupport(value: Double): SanityChecker.this.type

    Permalink
    Definition Classes
    SanityCheckerParams

  130. def setMinVariance(value: Double): SanityChecker.this.type

    Permalink
    Definition Classes
    SanityCheckerParams

  131. def setOutputFeatureName(name: String): SanityChecker.this.type

    Permalink
    Definition Classes
    OpPipelineStage

  132. def setProtectTextSharedHash(value: Boolean): SanityChecker.this.type

    Permalink
    Definition Classes
    SanityCheckerParams

  133. def setRemoveBadFeatures(value: Boolean): SanityChecker.this.type

    Permalink
    Definition Classes
    SanityCheckerParams

  134. def setRemoveFeatureGroup(value: Boolean): SanityChecker.this.type

    Permalink
    Definition Classes
    SanityCheckerParams

  135. def setSampleLowerLimit(value: Int): SanityChecker.this.type

    Permalink
    Definition Classes
    SanityCheckerParams

  136. def setSampleSeed(value: Long): SanityChecker.this.type

    Permalink
    Definition Classes
    SanityCheckerParams

  137. def setSampleUpperLimit(value: Int): SanityChecker.this.type

    Permalink
    Definition Classes
    SanityCheckerParams

  138. final def stageName: String

    Permalink
    Definition Classes
    OpPipelineStageBase

  139. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  140. def toString(): String

    Permalink
    Definition Classes
    Identifiable → AnyRef → Any

  141. final def transformSchema(schema: StructType): StructType

    Permalink
    Definition Classes
    OpPipelineStageBase

  142. def transformSchema(schema: StructType, logging: Boolean): StructType

    Permalink
    Attributes
    protected
    Definition Classes
    PipelineStage
    Annotations
    @DeveloperApi()

  143. implicit val tti1: scala.reflect.api.JavaUniverse.TypeTag[RealNN]

    Permalink
    Definition Classes
    BinaryEstimator

  144. implicit val tti2: scala.reflect.api.JavaUniverse.TypeTag[OPVector]

    Permalink
    Definition Classes
    BinaryEstimator

  145. implicit val ttiv1: scala.reflect.api.JavaUniverse.TypeTag[features.types.RealNN.Value]

    Permalink
    Definition Classes
    BinaryEstimator

  146. implicit val ttiv2: scala.reflect.api.JavaUniverse.TypeTag[Value]

    Permalink
    Definition Classes
    BinaryEstimator

  147. implicit val tto: scala.reflect.api.JavaUniverse.TypeTag[OPVector]

    Permalink
    Definition Classes
    BinaryEstimator → OpPipelineStage2

  148. implicit val ttov: scala.reflect.api.JavaUniverse.TypeTag[Value]

    Permalink
    Definition Classes
    BinaryEstimator → OpPipelineStage2

  149. implicit val tupleEncoder: Encoder[(features.types.RealNN.Value, Value)]

    Permalink
    Definition Classes
    BinaryEstimator

  150. val uid: String

    Permalink
    Definition Classes
    BinaryEstimator → Identifiable

  151. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  152. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  153. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  154. final def write: MLWriter

    Permalink
    Definition Classes
    OpPipelineStageBase → MLWritable

Inherited from AllowLabelAsInput[OPVector]

Inherited from SanityCheckerParams

Inherited from BinaryEstimator[RealNN, OPVector, OPVector]

Inherited from OpPipelineStage2[RealNN, OPVector, OPVector]

Inherited from HasIn2

Inherited from HasIn1

Inherited from OpPipelineStage[OPVector]

Inherited from OpPipelineStageBase

Inherited from MLWritable

Inherited from OpPipelineStageParams

Inherited from InputParams

Inherited from Estimator[BinaryModel[RealNN, OPVector, OPVector]]

Inherited from PipelineStage

Inherited from Logging

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

Ungrouped