MLUtils
Related Doc:
package util
Returns a new vector with 1.0 (bias) appended to the input vector.
Returns a new vector with 1.0 (bias) appended to the input vector.
Return a k element array of pairs of RDDs with the first element of each pair containing the training data, a complement of the validation data and the second element, the validation data, containing a unique 1/kth of the data.
Return a k element array of pairs of RDDs with the first element of each pair containing the training data, a complement of the validation data and the second element, the validation data, containing a unique 1/kth of the data. Where k=numFolds.
Loads labeled points saved using RDD[LabeledPoint].saveAsTextFile with the default number of
partitions.
Loads labeled points saved using RDD[LabeledPoint].saveAsTextFile with the default number of
partitions.
Loads labeled points saved using RDD[LabeledPoint].saveAsTextFile.
Loads labeled points saved using RDD[LabeledPoint].saveAsTextFile.
Spark context
file or directory path in any Hadoop-supported file system URI
min number of partitions
labeled points stored as an RDD[LabeledPoint]
Loads binary labeled data in the LIBSVM format into an RDD[LabeledPoint], with number of features determined automatically and the default number of partitions.
Loads binary labeled data in the LIBSVM format into an RDD[LabeledPoint], with number of features determined automatically and the default number of partitions.
Loads labeled data in the LIBSVM format into an RDD[LabeledPoint], with the default number of partitions.
Loads labeled data in the LIBSVM format into an RDD[LabeledPoint], with the default number of partitions.
Loads labeled data in the LIBSVM format into an RDD[LabeledPoint].
Loads labeled data in the LIBSVM format into an RDD[LabeledPoint]. The LIBSVM format is a text-based format used by LIBSVM and LIBLINEAR. Each line represents a labeled sparse feature vector using the following format:
label index1:value1 index2:value2 ...
where the indices are one-based and in ascending order. This method parses each line into a org.apache.spark.mllib.regression.LabeledPoint, where the feature indices are converted to zero-based.
Spark context
file or directory path in any Hadoop-supported file system URI
number of features, which will be determined from the input data if a nonpositive value is given. This is useful when the dataset is already split into multiple files and you want to load them separately, because some features may not present in certain files, which leads to inconsistent feature dimensions.
min number of partitions
labeled data stored as an RDD[LabeledPoint]
Loads vectors saved using RDD[Vector].saveAsTextFile with the default number of partitions.
Loads vectors saved using RDD[Vector].saveAsTextFile with the default number of partitions.
Loads vectors saved using RDD[Vector].saveAsTextFile.
Loads vectors saved using RDD[Vector].saveAsTextFile.
Spark context
file or directory path in any Hadoop-supported file system URI
min number of partitions
vectors stored as an RDD[Vector]
Save labeled data in LIBSVM format.
Save labeled data in LIBSVM format.
an RDD of LabeledPoint to be saved
directory to save the data
Load labeled data from a file.
Load labeled data from a file. The data format used here is L, f1 f2 ... where f1, f2 are feature values in Double and L is the corresponding label as Double.
SparkContext
Directory to the input data files.
An RDD of LabeledPoint. Each labeled point has two elements: the first element is the label, and the second element represents the feature values (an array of Double).
(Since version 1.0.1) Should use MLUtils.loadLabeledPoints instead.
(Since version 1.1.0) use method without multiclass argument, which no longer has effect
(Since version 1.1.0) use method without multiclass argument, which no longer has effect
(Since version 1.1.0) use method without multiclass argument, which no longer has effect
Save labeled data to a file.
Save labeled data to a file. The data format used here is L, f1 f2 ... where f1, f2 are feature values in Double and L is the corresponding label as Double.
An RDD of LabeledPoints containing data to be saved.
Directory to save the data.
(Since version 1.0.1) Should use RDD[LabeledPoint].saveAsTextFile instead.
Helper methods to load, save and pre-process data used in ML Lib.