Class/Object

org.apache.spark.ml.odkl

LinearDSVRGD

Related Docs: object LinearDSVRGD | package odkl

Permalink

class LinearDSVRGD extends DeVectorizedDSVRGD[LinearRegressionModel]

Single-label linear regresion with DSVRGD

Linear Supertypes
DeVectorizedDSVRGD[LinearRegressionModel], DSVRGD[LinearRegressionModel], HasCacheTrainData, HasTol, HasMaxIter, HasNetlibBlas, HasElasticNetParam, HasRegParam, HasLabelCol, HasFeaturesCol, HasPredictionCol, SummarizableEstimator[LinearRegressionModel], Estimator[LinearRegressionModel], PipelineStage, Logging, Params, Serializable, Serializable, Identifiable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. LinearDSVRGD
  2. DeVectorizedDSVRGD
  3. DSVRGD
  4. HasCacheTrainData
  5. HasTol
  6. HasMaxIter
  7. HasNetlibBlas
  8. HasElasticNetParam
  9. HasRegParam
  10. HasLabelCol
  11. HasFeaturesCol
  12. HasPredictionCol
  13. SummarizableEstimator
  14. Estimator
  15. PipelineStage
  16. Logging
  17. Params
  18. Serializable
  19. Serializable
  20. Identifiable
  21. AnyRef
  22. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new LinearDSVRGD()

    Permalink
  2. new LinearDSVRGD(uid: String)

    Permalink

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def $[T](param: Param[T]): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  4. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  5. def addGradient(weights: Matrix, features: DenseMatrix, labels: DenseMatrix, updateTerm: DenseMatrix, marginCache: DenseMatrix, lossCache: DenseVector): Unit

    Permalink

    For single instance and weights calculates gradient and loss.

    For single instance and weights calculates gradient and loss. Depending on direction adds gradient and loss to the accumulated data.

    weights

    Weights to evaluate gradient at

    features

    Featrues of instance to evaluate gradient at

    labels

    Labels of the instance to evaluate gradient at

    updateTerm

    Update term to store gradient at

    lossCache

    Loss vector to record resulting loss values.

    Attributes
    protected
    Definition Classes
    LinearDSVRGDDSVRGD
  6. def addL1Reg(l1regParam: Vector, weights: DenseMatrix, updateTerm: DenseMatrix, lossCache: DenseVector, skipRegFeature: Int): DenseMatrix

    Permalink
    Attributes
    protected
    Definition Classes
    DSVRGD
  7. def addL2Reg(l2regParam: Vector, weights: DenseMatrix, updateTerm: DenseMatrix, lossCache: DenseVector, skipRegFeature: Int): DenseMatrix

    Permalink

    Adds L2 regularization part to the gradient and loss.

    Adds L2 regularization part to the gradient and loss.

    Attributes
    protected
    Definition Classes
    DSVRGD
  8. def adjust(direction: Int, learningRates: DenseMatrix, updateTerm: DenseMatrix, weights: DenseMatrix): DenseMatrix

    Permalink
    Definition Classes
    DSVRGD
  9. def applyL1Shrinkage(regParam: Vector, weights: DenseMatrix, skipRegFeature: Int, notDegraded: Set[Int]): DenseMatrix

    Permalink

    Apply L1 shrinkage to the updated weights.

    Apply L1 shrinkage to the updated weights.

    Attributes
    protected
    Definition Classes
    DSVRGD
  10. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  11. def axpy(a: Double, x: Vector, y: Array[Double]): Unit

    Permalink
    Definition Classes
    HasNetlibBlas
  12. def axpy(a: Double, x: Array[Double], y: Array[Double]): Unit

    Permalink
    Definition Classes
    HasNetlibBlas
  13. def axpyCompensated(updateTerm: Array[Double], sum: Array[Double], compensator: Array[Double], y: Array[Double], t: Array[Double]): Unit

    Permalink
    Definition Classes
    DSVRGD
  14. def blas: BLAS

    Permalink
    Definition Classes
    HasNetlibBlas
  15. final val cacheTrainData: BooleanParam

    Permalink
    Definition Classes
    HasCacheTrainData
  16. final def clear(param: Param[_]): LinearDSVRGD.this.type

    Permalink
    Definition Classes
    Params
  17. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  18. val convergenceMode: Param[String]

    Permalink
    Definition Classes
    DSVRGD
  19. def copy(extra: ParamMap): DSVRGD[LinearRegressionModel]

    Permalink
    Definition Classes
    DSVRGDSummarizableEstimator → Estimator → PipelineStage → Params
  20. def copy(x: Array[Double], y: Array[Double]): Unit

    Permalink
    Definition Classes
    HasNetlibBlas
  21. def copyValues[T <: Params](to: T, extra: ParamMap): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  22. final def defaultCopy[T <: Params](extra: ParamMap): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  23. def dscal(a: Double, data: Array[Double]): Unit

    Permalink
    Definition Classes
    HasNetlibBlas
  24. final val elasticNetParam: DoubleParam

    Permalink
    Definition Classes
    HasElasticNetParam
  25. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  26. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  27. def evaluateL1Regularization(data: DataFrame, l1Scalar: Double, numLabels: Int): Vector

    Permalink

    Given L1 regularization config create a vector with per-label reg param (by default - constant).

    Given L1 regularization config create a vector with per-label reg param (by default - constant).

    Attributes
    protected
    Definition Classes
    DSVRGD
  28. def evaluateL2Regularization(data: DataFrame, l2Scalar: Double, numLabels: Int): Vector

    Permalink

    Given L2 regularization config create a vector with per-label reg param (by default - constant).

    Given L2 regularization config create a vector with per-label reg param (by default - constant).

    Attributes
    protected
    Definition Classes
    DSVRGD
  29. def explainParam(param: Param[_]): String

    Permalink
    Definition Classes
    Params
  30. def explainParams(): String

    Permalink
    Definition Classes
    Params
  31. def extractBlock(lossHistory: Array[CompactBuffer[Double]], dataset: DataFrame, names: Map[Int, String], sc: SparkContext): DataFrame

    Permalink
    Definition Classes
    DeVectorizedDSVRGDDSVRGD
  32. def extractLabelVectors(labelAttributeGroup: AttributeGroup, numLabels: Int, weights: Matrix): Map[String, Vector]

    Permalink

    Utility used to split weights matrice into label -> vector map

    Utility used to split weights matrice into label -> vector map

    Attributes
    protected
    Definition Classes
    DSVRGD
  33. def extractModel(labelAttributeGroup: AttributeGroup, numLabels: Int, weights: Matrix, dataset: DataFrame): LinearRegressionModel

    Permalink

    Given labels info and weights matrice create appropriate ML models.

    Given labels info and weights matrice create appropriate ML models.

    Attributes
    protected
    Definition Classes
    LinearDSVRGDDSVRGD
  34. final def extractParamMap(): ParamMap

    Permalink
    Definition Classes
    Params
  35. final def extractParamMap(extra: ParamMap): ParamMap

    Permalink
    Definition Classes
    Params
  36. def extractRow(label: Int, weights: Matrix): Vector

    Permalink

    Extracts a single row from a matrice.

    Extracts a single row from a matrice.

    Attributes
    protected
    Definition Classes
    DSVRGD
  37. def extractSummaryBlocks(lossHistory: Array[CompactBuffer[Double]], weightDiffHistory: Array[CompactBuffer[Double]], weightNormHistory: Array[CompactBuffer[Double]], dataset: DataFrame, labelAttributeGroup: AttributeGroup): Map[Block, DataFrame]

    Permalink

    Extracts summary blocks from iterations loss history.

    Extracts summary blocks from iterations loss history.

    Attributes
    protected
    Definition Classes
    DSVRGD
  38. def f2jBLAS: BLAS

    Permalink
    Definition Classes
    HasNetlibBlas
  39. final val featuresCol: Param[String]

    Permalink
    Definition Classes
    HasFeaturesCol
  40. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  41. def fit(dataset: Dataset[_]): LinearRegressionModel

    Permalink
    Definition Classes
    DeVectorizedDSVRGDDSVRGD → Estimator
  42. def fit(dataset: Dataset[_], paramMaps: Array[ParamMap]): Seq[LinearRegressionModel]

    Permalink
    Definition Classes
    Estimator
    Annotations
    @Since( "2.0.0" )
  43. def fit(dataset: Dataset[_], paramMap: ParamMap): LinearRegressionModel

    Permalink
    Definition Classes
    Estimator
    Annotations
    @Since( "2.0.0" )
  44. def fit(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): LinearRegressionModel

    Permalink
    Definition Classes
    Estimator
    Annotations
    @Since( "2.0.0" ) @varargs()
  45. def fullGradientAndLoss(l1regParam: Vector, l2regParam: Vector, localWeights: DenseMatrix, marginCache: DenseMatrix, lossCache: DenseVector, updateTerm: DenseMatrix, skipRegFeature: Int, features: DenseMatrix, labels: DenseMatrix): Any

    Permalink
    Definition Classes
    DSVRGD
  46. final def get[T](param: Param[T]): Option[T]

    Permalink
    Definition Classes
    Params
  47. final def getCacheTrainData: Boolean

    Permalink

    Definition Classes
    HasCacheTrainData
  48. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  49. final def getDefault[T](param: Param[T]): Option[T]

    Permalink
    Definition Classes
    Params
  50. final def getElasticNetParam: Double

    Permalink
    Definition Classes
    HasElasticNetParam
  51. final def getFeaturesCol: String

    Permalink
    Definition Classes
    HasFeaturesCol
  52. final def getLabelCol: String

    Permalink
    Definition Classes
    HasLabelCol
  53. final def getMaxIter: Int

    Permalink
    Definition Classes
    HasMaxIter
  54. def getNotConverged(activeLabels: Map[Int, Int], lossHistory: Array[CompactBuffer[Double]], weightDiffHistory: Array[CompactBuffer[Double]], weightNormHistory: Array[CompactBuffer[Double]], tolerance: Double): Array[Int]

    Permalink

    Extracts not converged labels based on actual and previous weights and on the loss history.

    Extracts not converged labels based on actual and previous weights and on the loss history.

    Attributes
    protected
    Definition Classes
    DSVRGD
  55. final def getOrDefault[T](param: Param[T]): T

    Permalink
    Definition Classes
    Params
  56. def getParam(paramName: String): Param[Any]

    Permalink
    Definition Classes
    Params
  57. final def getPredictionCol: String

    Permalink
    Definition Classes
    HasPredictionCol
  58. final def getRegParam: Double

    Permalink
    Definition Classes
    HasRegParam
  59. final def getTol: Double

    Permalink
    Definition Classes
    HasTol
  60. final def hasDefault[T](param: Param[T]): Boolean

    Permalink
    Definition Classes
    Params
  61. def hasParam(paramName: String): Boolean

    Permalink
    Definition Classes
    Params
  62. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  63. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  64. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  65. def initializeWeights(data: DataFrame, numLabels: Int, numFeatures: Int): Matrix

    Permalink
    Definition Classes
    DSVRGD
  66. final def isDefined(param: Param[_]): Boolean

    Permalink
    Definition Classes
    Params
  67. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  68. final def isSet(param: Param[_]): Boolean

    Permalink
    Definition Classes
    Params
  69. def isTraceEnabled(): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  70. final val labelCol: Param[String]

    Permalink
    Definition Classes
    HasLabelCol
  71. val lastIsIntercept: BooleanParam

    Permalink
    Definition Classes
    DSVRGD
  72. val learningRate: DoubleParam

    Permalink
    Definition Classes
    DSVRGD
  73. val localMinibatchSize: IntParam

    Permalink
    Definition Classes
    DSVRGD
  74. def log: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  75. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  76. def logDebug(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  77. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  78. def logError(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  79. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  80. def logInfo(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  81. def logName: String

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  82. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  83. def logTrace(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  84. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  85. def logWarning(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  86. def lossDifferenceForLabel(lossHistory: Array[CompactBuffer[Double]], label: Int): Double

    Permalink

    Evaluates loss difference simply as relative change

    Evaluates loss difference simply as relative change

    Definition Classes
    DSVRGD
  87. val lossIncreaseTolerance: DoubleParam

    Permalink
    Definition Classes
    DSVRGD
  88. final val maxIter: IntParam

    Permalink
    Definition Classes
    HasMaxIter
  89. def merge(labelsMap: Map[Int, Int], weights: Matrix, newWeights: DenseMatrix): DenseMatrix

    Permalink

    Merges weights from the new epoch with overal weights.

    Merges weights from the new epoch with overal weights. Dimensions of weights matrices might be different when part of labels are already converged and do not participate in descend.

    Attributes
    protected
    Definition Classes
    DSVRGD
  90. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  91. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  92. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  93. lazy val params: Array[Param[_]]

    Permalink
    Definition Classes
    Params
  94. final val predictionCol: Param[String]

    Permalink
    Definition Classes
    HasPredictionCol
  95. final val regParam: DoubleParam

    Permalink
    Definition Classes
    HasRegParam
  96. def relabel(activeLabels: Array[Int], labels: Vector): DenseVector

    Permalink

    Used to preserve only active (not yet converged) labels into a vector

    Used to preserve only active (not yet converged) labels into a vector

    Attributes
    protected
    Definition Classes
    DSVRGD
  97. def relabelMatrix(activeLabels: Array[Int], matrix: Matrix): Matrix

    Permalink

    Used to preserve only active (not yet converged) labels into a matrix

    Used to preserve only active (not yet converged) labels into a matrix

    Attributes
    protected
    Definition Classes
    DSVRGD
  98. final def set(paramPair: ParamPair[_]): LinearDSVRGD.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  99. final def set(param: String, value: Any): LinearDSVRGD.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  100. final def set[T](param: Param[T], value: T): LinearDSVRGD.this.type

    Permalink
    Definition Classes
    Params
  101. def setCacheTrainData(value: Boolean): LinearDSVRGD.this.type

    Permalink
    Definition Classes
    HasCacheTrainData
  102. def setConvergenceMode(value: String): LinearDSVRGD.this.type

    Permalink
    Definition Classes
    DSVRGD
  103. final def setDefault(paramPairs: ParamPair[_]*): LinearDSVRGD.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  104. final def setDefault[T](param: Param[T], value: T): LinearDSVRGD.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  105. def setElasticNetParam(value: Double): LinearDSVRGD.this.type

    Permalink
    Definition Classes
    DSVRGD
  106. def setLastIsIntercept(value: Boolean): LinearDSVRGD.this.type

    Permalink
    Definition Classes
    DSVRGD
  107. def setLearningRate(value: Double): LinearDSVRGD.this.type

    Permalink
    Definition Classes
    DSVRGD
  108. def setLocalMinibatchSize(value: Int): LinearDSVRGD.this.type

    Permalink
    Definition Classes
    DSVRGD
  109. def setMaxIter(value: Int): LinearDSVRGD.this.type

    Permalink
    Definition Classes
    DSVRGD
  110. def setRegParam(value: Double): LinearDSVRGD.this.type

    Permalink
    Definition Classes
    DSVRGD
  111. def setSlowDownFactor(value: Double): LinearDSVRGD.this.type

    Permalink
    Definition Classes
    DSVRGD
  112. def setSpeedUpFactor(value: Double): LinearDSVRGD.this.type

    Permalink
    Definition Classes
    DSVRGD
  113. def setTol(value: Double): LinearDSVRGD.this.type

    Permalink
    Definition Classes
    DSVRGD
  114. def singleStep(data: RDD[(Vector, DenseVector)], weights: Broadcast[Matrix], avgWeights: Broadcast[Matrix], avgGradient: Broadcast[Matrix], l1regParam: Vector, l2regParam: Vector, stepNum: Int, labelLearningRates: DenseVector): DistributedSgdState

    Permalink

    Single epoch of the descend

    Single epoch of the descend

    data

    Data with features and labels

    weights

    Weghts matrix to start with.

    avgWeights

    Average weights among walked during previous epoch.

    avgGradient

    Average gradient among seen during previous epoch.

    l1regParam

    Vector with the strength of L1 regularization (null if disabled)

    l2regParam

    Vector with the strength of L2 regularization (null if disabled)

    stepNum

    Number of epoch

    returns

    State with weights, averages and loss from this epoch

    Attributes
    protected
    Definition Classes
    DSVRGD
  115. val slowDownFactor: DoubleParam

    Permalink
    Definition Classes
    DSVRGD
  116. val speedUpFactor: DoubleParam

    Permalink
    Definition Classes
    DSVRGD
  117. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  118. def toDense(weights: Broadcast[Matrix]): DenseMatrix

    Permalink
    Definition Classes
    DSVRGD
  119. def toString(): String

    Permalink
    Definition Classes
    Identifiable → AnyRef → Any
  120. final val tol: DoubleParam

    Permalink
    Definition Classes
    HasTol
  121. def transformSchema(schema: StructType): StructType

    Permalink
    Definition Classes
    DSVRGD → PipelineStage
    Annotations
    @DeveloperApi()
  122. def transformSchema(schema: StructType, logging: Boolean): StructType

    Permalink
    Attributes
    protected
    Definition Classes
    PipelineStage
    Annotations
    @DeveloperApi()
  123. val uid: String

    Permalink
    Definition Classes
    LinearDSVRGDDeVectorizedDSVRGDDSVRGD → Identifiable
  124. def updateWeights(stepSize: Double, updateTerm: DenseMatrix, weights: DenseMatrix): Unit

    Permalink

    Updates the weights given update term and current value.

    Updates the weights given update term and current value.

    Attributes
    protected
    Definition Classes
    DSVRGD
  125. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  126. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  127. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  128. def weightNorm(newWeights: Matrix, label: Int, skipRegFeature: Int): Double

    Permalink

    Evaluates weight norm for a given label.

    Evaluates weight norm for a given label.

    newWeights

    Weights matrix

    label

    Label to evaluate weights

    returns

    Weights norm.

    Attributes
    protected
    Definition Classes
    DSVRGD
  129. def weightsDistanceForLabel(oldWeights: Matrix, newWeights: DenseMatrix, label: Int): Double

    Permalink

    Evaluates weight distance based on old and new weights images.

    Evaluates weight distance based on old and new weights images.

    oldWeights

    Weights from the previous epoch

    newWeights

    Weights from the current epoch.

    label

    Label to check for convergence.

    returns

    Distance between old and new weights.

    Attributes
    protected
    Definition Classes
    LinearDSVRGDDSVRGD

Inherited from DSVRGD[LinearRegressionModel]

Inherited from HasCacheTrainData

Inherited from HasTol

Inherited from HasMaxIter

Inherited from HasNetlibBlas

Inherited from HasElasticNetParam

Inherited from HasRegParam

Inherited from HasLabelCol

Inherited from HasFeaturesCol

Inherited from HasPredictionCol

Inherited from Estimator[LinearRegressionModel]

Inherited from PipelineStage

Inherited from Logging

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

getParam

Ungrouped