clustering

Type Members

1. class DistributedLDAModel extends LDAModel

   

   :: Experimental ::

   :: Experimental ::

   Distributed model fitted by LDA. This type of model is currently only produced by Expectation-Maximization (EM).

   This model stores the inferred topics, the full training dataset, and the topic distribution for each training document.

   Annotations

   @Since( "1.6.0" ) @Experimental()

2. class KMeans extends Estimator[KMeansModel] with KMeansParams with DefaultParamsWritable

   

   :: Experimental :: K-means clustering with support for k-means|| initialization proposed by Bahmani et al.

   :: Experimental :: K-means clustering with support for k-means|| initialization proposed by Bahmani et al.

   Annotations

   @Since( "1.5.0" ) @Experimental()

   See also

   Bahmani et al., Scalable k-means++.

3. class KMeansModel extends Model[KMeansModel] with KMeansParams with MLWritable

   

   :: Experimental :: Model fitted by KMeans.

   :: Experimental :: Model fitted by KMeans.

   Annotations

   @Since( "1.5.0" ) @Experimental()

4. class LDA extends Estimator[LDAModel] with LDAParams with DefaultParamsWritable

   

   :: Experimental ::

   :: Experimental ::

   Latent Dirichlet Allocation (LDA), a topic model designed for text documents.

   Terminology:

   • "term" = "word": an element of the vocabulary
   • "token": instance of a term appearing in a document
   • "topic": multinomial distribution over terms representing some concept
   • "document": one piece of text, corresponding to one row in the input data

   References:

   • Original LDA paper (journal version): Blei, Ng, and Jordan. "Latent Dirichlet Allocation." JMLR, 2003.

   Input data (featuresCol): LDA is given a collection of documents as input data, via the featuresCol parameter. Each document is specified as a Vector of length vocabSize, where each entry is the count for the corresponding term (word) in the document. Feature transformers such as org.apache.spark.ml.feature.Tokenizer and org.apache.spark.ml.feature.CountVectorizer can be useful for converting text to word count vectors.

   Annotations

   @Since( "1.6.0" ) @Experimental()

   See also

   Latent Dirichlet allocation (Wikipedia)

5. sealed abstract class LDAModel extends Model[LDAModel] with LDAParams with Logging with MLWritable

   

   :: Experimental :: Model fitted by LDA.

   :: Experimental :: Model fitted by LDA.

   Annotations

   @Since( "1.6.0" ) @Experimental()

6. class LocalLDAModel extends LDAModel

   

   :: Experimental ::

   :: Experimental ::

   Local (non-distributed) model fitted by LDA.

   This model stores the inferred topics only; it does not store info about the training dataset.

   Annotations

   @Since( "1.6.0" ) @Experimental()