Object

com.enriquegrodrigo.spark.crowd.methods

RaykarMulti

Related Doc: package methods

Permalink

object RaykarMulti

Provides functions for transforming an annotation dataset into a standard label dataset using the Raykar algorithm for multiclass

This algorithm only works with com.enriquegrodrigo.spark.crowd.types.MulticlassAnnotation annotation datasets

Example: 

  1. result: RaykarMultiModel = RaykarMulti(dataset, annotations)
Version

0.1

See also

Raykar, Vikas C., et al. "Learning from crowds." Journal of Machine Learning Research 11.Apr (2010): 1297-1322.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. RaykarMulti
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Type Members

  1. case class AnnotationWithClassProb(example: Long, clas: Int, prob: Double, annotator: Long, annotation: Int) extends Product with Serializable

    Permalink

    Dataset with both annotations and class probability estimated in the previous step, for obtaining the soft frequency matrices.

    Dataset with both annotations and class probability estimated in the previous step, for obtaining the soft frequency matrices.

    Version

    0.1

  2. class AnnotationsLikelihoodAggregator extends Aggregator[EStepEstimationPoint, (Double, Double), Double]

    Permalink

    Obtains the likelihood for each example given a class (grouping keys)

    Obtains the likelihood for each example given a class (grouping keys)

    Version

    0.1

  3. case class AnnotationsWithLogisticPrediction(example: Long, clas: Int, prediction: Double, annotator: Long, annotation: Int) extends Product with Serializable

    Permalink

    Annotations with logistic prediction information for EStep

    Annotations with logistic prediction information for EStep

    Version

    0.1

  4. case class AnnotatorClassCombination(annotator: Long, clas: Int, k: Int) extends Product with Serializable

    Permalink

    Combinations of annotator and classes for frequency calculation taking into account all combinations

    Combinations of annotator and classes for frequency calculation taking into account all combinations

    Version

    0.1

  5. case class AnnotatorClassFrequency(annotator: Long, clas: Int, frequency: Double) extends Product with Serializable

    Permalink

    Dataset with soft frequency of (annotator,c) in the annotations dataset Represents the denominator in the corresponding element of the precisions matrices.

    Dataset with soft frequency of (annotator,c) in the annotations dataset Represents the denominator in the corresponding element of the precisions matrices.

    Version

    0.1

  6. case class AnnotatorFrequency(annotator: Long, clas: Int, k: Int, frequency: Double) extends Product with Serializable

    Permalink

    Dataset with soft frequency of (annotator,c,k) in the annotations dataset.

    Dataset with soft frequency of (annotator,c,k) in the annotations dataset. Represents the numerator in the corresponding element of the precisions matrices.

    Version

    0.1

  7. class ClassFrequencyAggregator extends Aggregator[(AnnotatorClassCombination, AnnotationWithClassProb), Double, Double]

    Permalink

    Obtains the soft frequency of appearance of the key (j,c)

    Obtains the soft frequency of appearance of the key (j,c)

    Version

    0.1

  8. case class EStepEstimationPoint(example: Long, clas: Int, prediction: Double, annotator: Long, annotation: Int, annotationProb: Double) extends Product with Serializable

    Permalink

    EStep estimation point with information about annotation probability and the logistic prediction.

    EStep estimation point with information about annotation probability and the logistic prediction.

    Version

    0.1

  9. class FrequencyAggregator extends Aggregator[(AnnotatorClassCombination, AnnotationWithClassProb), Double, Double]

    Permalink

    Obtains the soft frequency of appearance of the key (j,c,k)

    Obtains the soft frequency of appearance of the key (j,c,k)

    Version

    0.1

  10. case class LikelihoodPoint(example: Long, clas: Int, mu: Double, annotationsLikelihood: Double) extends Product with Serializable

    Permalink

    Likelihood estimation point with annotation likelihood as well as the true class estimation form E Step

    Likelihood estimation point with annotation likelihood as well as the true class estimation form E Step

    Version

    0.1

  11. case class LogisticAnnotatorParams(a: Double, b: Double) extends Product with Serializable

    Permalink

    Logistic Annotator params for the LogisticParams aggregator

    Logistic Annotator params for the LogisticParams aggregator

    Version

    0.1

  12. case class LogisticMultiPrediction(example: Long, clas: Int, prob: Double) extends Product with Serializable

    Permalink

    Logistic prediction for the full multiclass problem

    Logistic prediction for the full multiclass problem

    Version

    0.1

  13. class LogisticParamAggregator extends Aggregator[(MulticlassAnnotation, DiscreteAnnotatorPrecision), LogisticAnnotatorParams, LogisticAnnotatorParams]

    Permalink

    Obtains the soft frequency of appearance of the key (j,c)

    Obtains the soft frequency of appearance of the key (j,c)

    Version

    0.1

  14. case class LogisticParams(example: Long, a: Double, b: Double) extends Product with Serializable

    Permalink

    Aggregation of annotator parameters for each example in the one vs all approach for logistic regression.

    Aggregation of annotator parameters for each example in the one vs all approach for logistic regression.

    Version

    0.1

  15. case class LogisticPrediction(example: Long, prob: Double) extends Product with Serializable

    Permalink

    Logistic prediction for the one vs all approach

    Logistic prediction for the one vs all approach

    Version

    0.1

  16. case class MuWithLogisticParams(example: Long, mu: Double, a: Double, b: Double) extends Product with Serializable

    Permalink

    Mu estimate with logistic params for the example

    Mu estimate with logistic params for the example

    Version

    0.1

  17. case class Normalizer(example: Long, norm: Double) extends Product with Serializable

    Permalink

    Normalizer for logistic predictions

    Normalizer for logistic predictions

    Version

    0.1

  18. class RaykarMultiGradient extends Gradient

    Permalink

    Computes the gradient for the SGD algorithm

  19. case class RaykarMultiPartialModel(dataset: DataFrame, annotations: Dataset[MulticlassAnnotation], mu: Dataset[MulticlassSoftProb], annotatorPrecision: Dataset[DiscreteAnnotatorPrecision], logisticWeights: Array[Array[Double]], logisticPrediction: Dataset[LogisticMultiPrediction], annotationsLikelihood: Dataset[MulticlassSoftProb], annotatorPriorMatrix: Broadcast[Array[Array[Array[Double]]]], weightsPriorMatrix: Array[Broadcast[Array[Array[Double]]]], likelihood: Double, improvement: Double, annotatorClassCombination: Dataset[AnnotatorClassCombination], nFeatures: Int, nClasses: Int, nAnnotators: Int) extends Product with Serializable

    Permalink

    Partial object get data from one step to another.

    Partial object get data from one step to another.

    Version

    0.1

  20. class RaykarMultiUpdater extends Updater

    Permalink

    Computes updater for the SGD algorithm.

    Computes updater for the SGD algorithm. Adds the regularization priors.

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  4. def apply(dataset: DataFrame, annDataset: Dataset[MulticlassAnnotation], eMIters: Int = 3, eMThreshold: Double = 0.001, gradIters: Int = 100, gradThreshold: Double = 0.1, gradLearning: Double = 0.1, k_prior: Option[Array[Array[Array[Double]]]] = None, w_prior: Option[Array[Array[Array[Double]]]] = None): RaykarMultiModel

    Permalink

    Applies the learning algorithm

    Applies the learning algorithm

    dataset

    the dataset with feature vectors.

    annDataset

    the dataset with the annotations.

    gradIters

    maximum number of iterations for the GradientDescent algorithm

    gradThreshold

    threshold for the log likelihood variability for the gradient descent algorithm

    gradLearning

    learning rate for the gradient descent algorithm

    k_prior

    prior (Dirichlet distribution hyperparameters) for the estimation of the probability that an annotator correctly a class given another

    w_prior

    prior for the weights of the logistic regression model

    returns

    com.enriquegrodrigo.spark.crowd.types.RaykarBinaryModel

    Version

    0.1

  5. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  6. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  7. def computePointLoss(mui: Double, pi: Double, ai: Double, bi: Double): Double

    Permalink

    Computes the negative likelihood of a point (loss)

  8. def computeSigmoid(x: Array[Double], w: Array[Double]): Double

    Permalink

    Computes the logistic function for a data point

  9. def eStep(model: RaykarMultiPartialModel): RaykarMultiPartialModel

    Permalink

    E Step of the EM algorithm.

    E Step of the EM algorithm.

    Version

    0.1

  10. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  11. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  12. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  13. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  14. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  15. def initialization(dataset: DataFrame, annotatorData: Dataset[MulticlassAnnotation], k_prior: Option[Array[Array[Array[Double]]]], w_prior: Option[Array[Array[Array[Double]]]]): RaykarMultiPartialModel

    Permalink

    Initialize the parameters.

    Initialize the parameters. First ground truth estimation is done using the majority voting algorithm

    Version

    0.1

  16. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  17. def logLikelihood(model: RaykarMultiPartialModel): RaykarMultiPartialModel

    Permalink

    Obtains the likelihood of the partial model.

    Obtains the likelihood of the partial model.

    Version

    0.1

  18. def mStep(model: RaykarMultiPartialModel, gradIters: Int, gradThreshold: Double, gradLearning: Double): RaykarMultiPartialModel

    Permalink

    M Step of the EM algorithm.

    M Step of the EM algorithm.

    Version

    0.1

  19. def matMult(mat: Array[Array[Double]], v: Array[Double]): Array[Double]

    Permalink

    Matrix multiplication (TODO: improving using libraries)

  20. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  21. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  22. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  23. def step(gradIters: Int, gradThreshold: Double, gradLearning: Double)(model: RaykarMultiPartialModel, i: Int): RaykarMultiPartialModel

    Permalink

    Step of the iterative algorithm

    Step of the iterative algorithm

    Version

    0.1

  24. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  25. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any

  26. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  27. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  28. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from AnyRef

Inherited from Any

Ungrouped