Class/Object

org.apache.spark.ml.odkl

LogisticMatrixDSVRGD

Related Docs: object LogisticMatrixDSVRGD | package odkl

Permalink

class LogisticMatrixDSVRGD extends DSVRGD[LinearCombinationModel[LogisticRegressionModel]]

Multi-label logistic regresion with DSVRGD

Linear Supertypes
DSVRGD[LinearCombinationModel[LogisticRegressionModel]], HasCacheTrainData, HasTol, HasMaxIter, HasNetlibBlas, HasElasticNetParam, HasRegParam, HasLabelCol, HasFeaturesCol, HasPredictionCol, SummarizableEstimator[LinearCombinationModel[LogisticRegressionModel]], Estimator[LinearCombinationModel[LogisticRegressionModel]], PipelineStage, Logging, Params, Serializable, Serializable, Identifiable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. LogisticMatrixDSVRGD
  2. DSVRGD
  3. HasCacheTrainData
  4. HasTol
  5. HasMaxIter
  6. HasNetlibBlas
  7. HasElasticNetParam
  8. HasRegParam
  9. HasLabelCol
  10. HasFeaturesCol
  11. HasPredictionCol
  12. SummarizableEstimator
  13. Estimator
  14. PipelineStage
  15. Logging
  16. Params
  17. Serializable
  18. Serializable
  19. Identifiable
  20. AnyRef
  21. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new LogisticMatrixDSVRGD()

    Permalink
  2. new LogisticMatrixDSVRGD(uid: String)

    Permalink

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def $[T](param: Param[T]): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  4. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  5. def addGradient(weights: Matrix, features: DenseMatrix, labels: DenseMatrix, updateTerm: DenseMatrix, marginCache: DenseMatrix, lossCache: DenseVector): Unit

    Permalink

    For single instance and weights calculates gradient and loss.

    For single instance and weights calculates gradient and loss. Depending on direction adds gradient and loss to the accumulated data.

    weights

    Weights to evaluate gradient at

    features

    Featrues of instance to evaluate gradient at

    labels

    Labels of the instance to evaluate gradient at

    updateTerm

    Update term to store gradient at

    lossCache

    Loss vector to record resulting loss values.

    Attributes
    protected
    Definition Classes
    LogisticMatrixDSVRGDDSVRGD
  6. def addL1Reg(l1regParam: Vector, weights: DenseMatrix, updateTerm: DenseMatrix, lossCache: DenseVector, skipRegFeature: Int): DenseMatrix

    Permalink
    Attributes
    protected
    Definition Classes
    DSVRGD
  7. def addL2Reg(l2regParam: Vector, weights: DenseMatrix, updateTerm: DenseMatrix, lossCache: DenseVector, skipRegFeature: Int): DenseMatrix

    Permalink

    Adds L2 regularization part to the gradient and loss.

    Adds L2 regularization part to the gradient and loss.

    Attributes
    protected
    Definition Classes
    DSVRGD
  8. def adjust(direction: Int, learningRates: DenseMatrix, updateTerm: DenseMatrix, weights: DenseMatrix): DenseMatrix

    Permalink
    Definition Classes
    DSVRGD
  9. def applyL1Shrinkage(regParam: Vector, weights: DenseMatrix, skipRegFeature: Int, notDegraded: Set[Int]): DenseMatrix

    Permalink

    Apply L1 shrinkage to the updated weights.

    Apply L1 shrinkage to the updated weights.

    Attributes
    protected
    Definition Classes
    DSVRGD
  10. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  11. def axpy(a: Double, x: Vector, y: Array[Double]): Unit

    Permalink
    Definition Classes
    HasNetlibBlas
  12. def axpy(a: Double, x: Array[Double], y: Array[Double]): Unit

    Permalink
    Definition Classes
    HasNetlibBlas
  13. def axpyCompensated(updateTerm: Array[Double], sum: Array[Double], compensator: Array[Double], y: Array[Double], t: Array[Double]): Unit

    Permalink
    Definition Classes
    DSVRGD
  14. def blas: BLAS

    Permalink
    Definition Classes
    HasNetlibBlas
  15. final val cacheTrainData: BooleanParam

    Permalink
    Definition Classes
    HasCacheTrainData
  16. final def clear(param: Param[_]): LogisticMatrixDSVRGD.this.type

    Permalink
    Definition Classes
    Params
  17. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  18. val convergenceMode: Param[String]

    Permalink
    Definition Classes
    DSVRGD
  19. def copy(extra: ParamMap): DSVRGD[LinearCombinationModel[LogisticRegressionModel]]

    Permalink
    Definition Classes
    DSVRGDSummarizableEstimator → Estimator → PipelineStage → Params
  20. def copy(x: Array[Double], y: Array[Double]): Unit

    Permalink
    Definition Classes
    HasNetlibBlas
  21. def copyValues[T <: Params](to: T, extra: ParamMap): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  22. final def defaultCopy[T <: Params](extra: ParamMap): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  23. def dscal(a: Double, data: Array[Double]): Unit

    Permalink
    Definition Classes
    HasNetlibBlas
  24. final val elasticNetParam: DoubleParam

    Permalink
    Definition Classes
    HasElasticNetParam
  25. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  26. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  27. def evaluateL1Regularization(data: DataFrame, l1Scalar: Double, numLabels: Int): Vector

    Permalink

    Given L1 regularization config create a vector with per-label reg param (by default - constant).

    Given L1 regularization config create a vector with per-label reg param (by default - constant).

    Attributes
    protected
    Definition Classes
    LogisticMatrixDSVRGDDSVRGD
  28. def evaluateL2Regularization(data: DataFrame, l2Scalar: Double, numLabels: Int): Vector

    Permalink

    Given L2 regularization config create a vector with per-label reg param (by default - constant).

    Given L2 regularization config create a vector with per-label reg param (by default - constant).

    Attributes
    protected
    Definition Classes
    DSVRGD
  29. def explainParam(param: Param[_]): String

    Permalink
    Definition Classes
    Params
  30. def explainParams(): String

    Permalink
    Definition Classes
    Params
  31. def extractBlock(lossHistory: Array[CompactBuffer[Double]], dataset: DataFrame, names: Map[Int, String], sc: SparkContext): DataFrame

    Permalink
    Definition Classes
    DSVRGD
  32. def extractLabelVectors(labelAttributeGroup: AttributeGroup, numLabels: Int, weights: Matrix): Map[String, Vector]

    Permalink

    Utility used to split weights matrice into label -> vector map

    Utility used to split weights matrice into label -> vector map

    Attributes
    protected
    Definition Classes
    DSVRGD
  33. def extractModel(labelAttributeGroup: AttributeGroup, numLabels: Int, weights: Matrix, dataset: DataFrame): LinearCombinationModel[LogisticRegressionModel]

    Permalink

    Given labels info and weights matrice create appropriate ML models.

    Given labels info and weights matrice create appropriate ML models.

    Attributes
    protected
    Definition Classes
    LogisticMatrixDSVRGDDSVRGD
  34. final def extractParamMap(): ParamMap

    Permalink
    Definition Classes
    Params
  35. final def extractParamMap(extra: ParamMap): ParamMap

    Permalink
    Definition Classes
    Params
  36. def extractRow(label: Int, weights: Matrix): Vector

    Permalink

    Extracts a single row from a matrice.

    Extracts a single row from a matrice.

    Attributes
    protected
    Definition Classes
    DSVRGD
  37. def extractSummaryBlocks(lossHistory: Array[CompactBuffer[Double]], weightDiffHistory: Array[CompactBuffer[Double]], weightNormHistory: Array[CompactBuffer[Double]], dataset: DataFrame, labelAttributeGroup: AttributeGroup): Map[Block, DataFrame]

    Permalink

    Extracts summary blocks from iterations loss history.

    Extracts summary blocks from iterations loss history.

    Attributes
    protected
    Definition Classes
    DSVRGD
  38. def f2jBLAS: BLAS

    Permalink
    Definition Classes
    HasNetlibBlas
  39. final val featuresCol: Param[String]

    Permalink
    Definition Classes
    HasFeaturesCol
  40. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  41. def fit(dataset: Dataset[_]): LinearCombinationModel[LogisticRegressionModel]

    Permalink
    Definition Classes
    DSVRGD → Estimator
  42. def fit(dataset: Dataset[_], paramMaps: Array[ParamMap]): Seq[LinearCombinationModel[LogisticRegressionModel]]

    Permalink
    Definition Classes
    Estimator
    Annotations
    @Since( "2.0.0" )
  43. def fit(dataset: Dataset[_], paramMap: ParamMap): LinearCombinationModel[LogisticRegressionModel]

    Permalink
    Definition Classes
    Estimator
    Annotations
    @Since( "2.0.0" )
  44. def fit(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): LinearCombinationModel[LogisticRegressionModel]

    Permalink
    Definition Classes
    Estimator
    Annotations
    @Since( "2.0.0" ) @varargs()
  45. def fullGradientAndLoss(l1regParam: Vector, l2regParam: Vector, localWeights: DenseMatrix, marginCache: DenseMatrix, lossCache: DenseVector, updateTerm: DenseMatrix, skipRegFeature: Int, features: DenseMatrix, labels: DenseMatrix): Any

    Permalink
    Definition Classes
    DSVRGD
  46. final def get[T](param: Param[T]): Option[T]

    Permalink
    Definition Classes
    Params
  47. final def getCacheTrainData: Boolean

    Permalink

    Definition Classes
    HasCacheTrainData
  48. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  49. final def getDefault[T](param: Param[T]): Option[T]

    Permalink
    Definition Classes
    Params
  50. final def getElasticNetParam: Double

    Permalink
    Definition Classes
    HasElasticNetParam
  51. final def getFeaturesCol: String

    Permalink
    Definition Classes
    HasFeaturesCol
  52. final def getLabelCol: String

    Permalink
    Definition Classes
    HasLabelCol
  53. final def getMaxIter: Int

    Permalink
    Definition Classes
    HasMaxIter
  54. def getNotConverged(activeLabels: Map[Int, Int], lossHistory: Array[CompactBuffer[Double]], weightDiffHistory: Array[CompactBuffer[Double]], weightNormHistory: Array[CompactBuffer[Double]], tolerance: Double): Array[Int]

    Permalink

    Extracts not converged labels based on actual and previous weights and on the loss history.

    Extracts not converged labels based on actual and previous weights and on the loss history.

    Attributes
    protected
    Definition Classes
    DSVRGD
  55. final def getOrDefault[T](param: Param[T]): T

    Permalink
    Definition Classes
    Params
  56. def getParam(paramName: String): Param[Any]

    Permalink
    Definition Classes
    Params
  57. final def getPredictionCol: String

    Permalink
    Definition Classes
    HasPredictionCol
  58. final def getRegParam: Double

    Permalink
    Definition Classes
    HasRegParam
  59. final def getTol: Double

    Permalink
    Definition Classes
    HasTol
  60. final def hasDefault[T](param: Param[T]): Boolean

    Permalink
    Definition Classes
    Params
  61. def hasParam(paramName: String): Boolean

    Permalink
    Definition Classes
    Params
  62. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  63. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  64. def initializeWeights(data: DataFrame, numLabels: Int, numFeatures: Int): Matrix

    Permalink
    Definition Classes
    LogisticMatrixDSVRGDDSVRGD
  65. final def isDefined(param: Param[_]): Boolean

    Permalink
    Definition Classes
    Params
  66. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  67. final def isSet(param: Param[_]): Boolean

    Permalink
    Definition Classes
    Params
  68. def isTraceEnabled(): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  69. final val labelCol: Param[String]

    Permalink
    Definition Classes
    HasLabelCol
  70. val lastIsIntercept: BooleanParam

    Permalink
    Definition Classes
    DSVRGD
  71. val learningRate: DoubleParam

    Permalink
    Definition Classes
    DSVRGD
  72. val localMinibatchSize: Param[Int]

    Permalink
    Definition Classes
    DSVRGD
  73. def log: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  74. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  75. def logDebug(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  76. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  77. def logError(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  78. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  79. def logInfo(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  80. def logName: String

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  81. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  82. def logTrace(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  83. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  84. def logWarning(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  85. def lossDifferenceForLabel(lossHistory: Array[CompactBuffer[Double]], label: Int): Double

    Permalink

    Evaluates loss difference simply as relative change

    Evaluates loss difference simply as relative change

    Definition Classes
    DSVRGD
  86. val lossIncreaseTolerance: DoubleParam

    Permalink
    Definition Classes
    DSVRGD
  87. final val maxIter: IntParam

    Permalink
    Definition Classes
    HasMaxIter
  88. def merge(labelsMap: Map[Int, Int], weights: Matrix, newWeights: DenseMatrix): DenseMatrix

    Permalink

    Merges weights from the new epoch with overal weights.

    Merges weights from the new epoch with overal weights. Dimensions of weights matrices might be different when part of labels are already converged and do not participate in descend.

    Attributes
    protected
    Definition Classes
    DSVRGD
  89. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  90. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  91. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  92. lazy val params: Array[Param[_]]

    Permalink
    Definition Classes
    Params
  93. final val predictionCol: Param[String]

    Permalink
    Definition Classes
    HasPredictionCol
  94. final val regParam: DoubleParam

    Permalink
    Definition Classes
    HasRegParam
  95. def relabel(activeLabels: Array[Int], labels: Vector): DenseVector

    Permalink

    Used to preserve only active (not yet converged) labels into a vector

    Used to preserve only active (not yet converged) labels into a vector

    Attributes
    protected
    Definition Classes
    DSVRGD
  96. def relabelMatrix(activeLabels: Array[Int], matrix: Matrix): Matrix

    Permalink

    Used to preserve only active (not yet converged) labels into a matrix

    Used to preserve only active (not yet converged) labels into a matrix

    Attributes
    protected
    Definition Classes
    DSVRGD
  97. final def set(paramPair: ParamPair[_]): LogisticMatrixDSVRGD.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  98. final def set(param: String, value: Any): LogisticMatrixDSVRGD.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  99. final def set[T](param: Param[T], value: T): LogisticMatrixDSVRGD.this.type

    Permalink
    Definition Classes
    Params
  100. def setCacheTrainData(value: Boolean): LogisticMatrixDSVRGD.this.type

    Permalink
    Definition Classes
    HasCacheTrainData
  101. def setConvergenceMode(value: String): LogisticMatrixDSVRGD.this.type

    Permalink
    Definition Classes
    DSVRGD
  102. final def setDefault(paramPairs: ParamPair[_]*): LogisticMatrixDSVRGD.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  103. final def setDefault[T](param: Param[T], value: T): LogisticMatrixDSVRGD.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  104. def setElasticNetParam(value: Double): LogisticMatrixDSVRGD.this.type

    Permalink
    Definition Classes
    DSVRGD
  105. def setLastIsIntercept(value: Boolean): LogisticMatrixDSVRGD.this.type

    Permalink
    Definition Classes
    DSVRGD
  106. def setLearningRate(value: Double): LogisticMatrixDSVRGD.this.type

    Permalink
    Definition Classes
    DSVRGD
  107. def setLocalMinibatchSize(value: Int): LogisticMatrixDSVRGD.this.type

    Permalink
    Definition Classes
    DSVRGD
  108. def setMaxIter(value: Int): LogisticMatrixDSVRGD.this.type

    Permalink
    Definition Classes
    DSVRGD
  109. def setRegParam(value: Double): LogisticMatrixDSVRGD.this.type

    Permalink
    Definition Classes
    DSVRGD
  110. def setSlowDownFactor(value: Double): LogisticMatrixDSVRGD.this.type

    Permalink
    Definition Classes
    DSVRGD
  111. def setSpeedUpFactor(value: Double): LogisticMatrixDSVRGD.this.type

    Permalink
    Definition Classes
    DSVRGD
  112. def setTol(value: Double): LogisticMatrixDSVRGD.this.type

    Permalink
    Definition Classes
    DSVRGD
  113. def singleStep(data: RDD[(Vector, DenseVector)], weights: Broadcast[Matrix], avgWeights: Broadcast[Matrix], avgGradient: Broadcast[Matrix], l1regParam: Vector, l2regParam: Vector, stepNum: Int, labelLearningRates: DenseVector): DistributedSgdState

    Permalink

    Single epoch of the descend

    Single epoch of the descend

    data

    Data with features and labels

    weights

    Weghts matrix to start with.

    avgWeights

    Average weights among walked during previous epoch.

    avgGradient

    Average gradient among seen during previous epoch.

    l1regParam

    Vector with the strength of L1 regularization (null if disabled)

    l2regParam

    Vector with the strength of L2 regularization (null if disabled)

    stepNum

    Number of epoch

    returns

    State with weights, averages and loss from this epoch

    Attributes
    protected
    Definition Classes
    DSVRGD
  114. val slowDownFactor: DoubleParam

    Permalink
    Definition Classes
    DSVRGD
  115. val speedUpFactor: DoubleParam

    Permalink
    Definition Classes
    DSVRGD
  116. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  117. def toDense(weights: Broadcast[Matrix]): DenseMatrix

    Permalink
    Definition Classes
    DSVRGD
  118. def toString(): String

    Permalink
    Definition Classes
    Identifiable → AnyRef → Any
  119. final val tol: DoubleParam

    Permalink
    Definition Classes
    HasTol
  120. def transformSchema(schema: StructType): StructType

    Permalink
    Definition Classes
    DSVRGD → PipelineStage
    Annotations
    @DeveloperApi()
  121. def transformSchema(schema: StructType, logging: Boolean): StructType

    Permalink
    Attributes
    protected
    Definition Classes
    PipelineStage
    Annotations
    @DeveloperApi()
  122. val uid: String

    Permalink
    Definition Classes
    LogisticMatrixDSVRGDDSVRGD → Identifiable
  123. def updateWeights(stepSize: Double, updateTerm: DenseMatrix, weights: DenseMatrix): Unit

    Permalink

    Updates the weights given update term and current value.

    Updates the weights given update term and current value.

    Attributes
    protected
    Definition Classes
    DSVRGD
  124. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  125. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  126. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  127. def weightNorm(newWeights: Matrix, label: Int, skipRegFeature: Int): Double

    Permalink

    Evaluates weight norm for a given label.

    Evaluates weight norm for a given label.

    newWeights

    Weights matrix

    label

    Label to evaluate weights

    returns

    Weights norm.

    Attributes
    protected
    Definition Classes
    DSVRGD
  128. def weightsDistanceForLabel(oldWeights: Matrix, newWeights: DenseMatrix, label: Int): Double

    Permalink

    Evaluates weight distance based on old and new weights images.

    Evaluates weight distance based on old and new weights images.

    oldWeights

    Weights from the previous epoch

    newWeights

    Weights from the current epoch.

    label

    Label to check for convergence.

    returns

    Distance between old and new weights.

    Attributes
    protected
    Definition Classes
    LogisticMatrixDSVRGDDSVRGD

Inherited from HasCacheTrainData

Inherited from HasTol

Inherited from HasMaxIter

Inherited from HasNetlibBlas

Inherited from HasElasticNetParam

Inherited from HasRegParam

Inherited from HasLabelCol

Inherited from HasFeaturesCol

Inherited from HasPredictionCol

Inherited from Estimator[LinearCombinationModel[LogisticRegressionModel]]

Inherited from PipelineStage

Inherited from Logging

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

getParam

Ungrouped