Object

io.github.mandar2812.dynaml

DynaMLPipe

Related Doc: package dynaml

Permalink

object DynaMLPipe

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. DynaMLPipe
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  4. def GPRegressionTest[T <: AbstractGPRegressionModel[Seq[(DenseVector[Double], Double)], DenseVector[Double]]](model: T): SideEffectPipe[(Stream[(DenseVector[Double], Double)], (DenseVector[Double], DenseVector[Double]))]

    Permalink
  5. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  6. def baggingRDD[T](proportion: Double, nBags: Int): DataPipe[RDD[T], IndexedSeq[RDD[T]]]

    Permalink

    Returns a pipeline which performs a bagging based sub-sampling of an Apache Spark RDD of T.

    Returns a pipeline which performs a bagging based sub-sampling of an Apache Spark RDD of T.

    proportion

    The sampling proportion between 0 and 1

    nBags

    The number of bags to generate.

  7. def baggingStream[T](proportion: Double, nBags: Int): DataPipe[Stream[T], Stream[Stream[T]]]

    Permalink

    Returns a pipeline which performs a bagging based sub-sampling of a stream of T.

    Returns a pipeline which performs a bagging based sub-sampling of a stream of T.

    proportion

    The sampling proportion between 0 and 1

    nBags

    The number of bags to generate.

  8. val breezeDVReplicationEncoder: (Int) ⇒ Encoder[DenseVector[Double], Array[DenseVector[Double]]]

    Permalink

    Creates an Encoder which replicates a DenseVector instance n times.

  9. val breezeDVSplitEncoder: (Int) ⇒ Encoder[DenseVector[Double], Array[DenseVector[Double]]]

    Permalink

    Creates an Encoder which can split DenseVector instances into uniform splits and put them back together.

  10. def calculateGaussianScales(standardize: Boolean = true): DataPipe[Stream[(DenseVector[Double], DenseVector[Double])], (Stream[(DenseVector[Double], DenseVector[Double])], (GaussianScaler, GaussianScaler))]

    Permalink

    Returns a pipe which takes a data set and calculates the mean and standard deviation of each dimension.

    Returns a pipe which takes a data set and calculates the mean and standard deviation of each dimension.

    standardize

    Set to true if one wants the standardized data and false if one does wants the original data with the GaussianScaler instances.

  11. def calculateMVGaussianScales(standardize: Boolean = true): DataPipe[Stream[(DenseVector[Double], DenseVector[Double])], (Stream[(DenseVector[Double], DenseVector[Double])], (MVGaussianScaler, MVGaussianScaler))]

    Permalink

    Multivariate version of calculateGaussianScales

    Multivariate version of calculateGaussianScales

    standardize

    Set to true if one wants the standardized data and false if one does wants the original data with the MVGaussianScaler instances.

  12. def calculateMeanScales(standardize: Boolean = true): DataPipe[Stream[(DenseVector[Double], DenseVector[Double])], (Stream[(DenseVector[Double], DenseVector[Double])], (MeanScaler, MeanScaler))]

    Permalink

    Returns a pipe which takes a data set and mean centers it.

    Returns a pipe which takes a data set and mean centers it.

    standardize

    Set to true if one wants the standardized data and false if one does wants the original data with the MeanScaler instances.

  13. def calculateMinMaxScales(standardize: Boolean = true): DataPipe[Stream[(DenseVector[Double], DenseVector[Double])], (Stream[(DenseVector[Double], DenseVector[Double])], (MinMaxScaler, MinMaxScaler))]

    Permalink

    Returns a pipe which takes a data set and calculates the minimum and maximum of each dimension.

    Returns a pipe which takes a data set and calculates the minimum and maximum of each dimension.

    standardize

    Set to true if one wants the standardized data and false if one does wants the original data with the MinMaxScaler instances.

  14. def calculatePCAScales(standardize: Boolean = true): DataPipe[Stream[(DenseVector[Double], DenseVector[Double])], (Stream[(DenseVector[Double], DenseVector[Double])], (PCAScaler, MVGaussianScaler))]

    Permalink

    Returns a pipe which performs PCA on data features and gaussian scaling on data targets

    Returns a pipe which performs PCA on data features and gaussian scaling on data targets

    standardize

    Set to true if one wants the standardized data and false if one does wants the original data with the MVGaussianScaler instances.

  15. def calculatePCAScalesFeatures(standardize: Boolean = true): DataPipe[Stream[DenseVector[Double]], (Stream[DenseVector[Double]], PCAScaler)]

    Permalink

    Returns a pipe which performs PCA on data features and gaussian scaling on data targets

    Returns a pipe which performs PCA on data features and gaussian scaling on data targets

    standardize

    Set to true if one wants the standardized data and false if one does wants the original data with the MVGaussianScaler instances.

  16. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  17. def csvToRDF(df: String, sep: Char)(implicit renjin: RenjinScriptEngine): DataPipe[String, ListVector]

    Permalink

    Read a csv text file and store it in a R data frame.

    Read a csv text file and store it in a R data frame.

    df

    The name of the data frame variable

    sep

    Separation character in the csv file

    returns

    A DataPipe instance which takes as input a file name and returns a renjin ListVector instance and stores data frame in the variable nameed as df.

  18. val deltaOperation: (Int, Int) ⇒ DataPipe[Stream[(Double, Double)], Stream[(DenseVector[Double], Double)]]

    Permalink

    Inorder to generate features for auto-regressive models, one needs to construct sliding windows in time.

    Inorder to generate features for auto-regressive models, one needs to construct sliding windows in time. This function takes two parameters

    deltaT: the auto-regressive order timelag: the time lag after which the windowing is conducted.

    E.g

    Let deltaT = 2 and timelag = 1

    This pipe will take stream data of the form (t, Value_t)

    and output a stream which looks like

    (t, Vector(Value_t-2, Value_t-3))

  19. val deltaOperationARX: (List[Int]) ⇒ DataPipe[Stream[(Double, DenseVector[Double])], Stream[(DenseVector[Double], Double)]]

    Permalink

    The vector ARX version of DynaMLPipe.deltaOperation

  20. val deltaOperationVec: (Int) ⇒ DataPipe[Stream[(Double, DenseVector[Double])], Stream[(DenseVector[Double], Double)]]

    Permalink

    The vector version of DynaMLPipe.deltaOperation

  21. val dropHead: DataPipe[Stream[String], Stream[String]]

    Permalink

    Drop the first element of a Stream of String

  22. def duplicate[Source, Destination](pipe: DataPipe[Source, Destination]): ParallelPipe[Source, Destination, Source, Destination]

    Permalink

    Takes a base pipe and creates a parallel pipe by duplicating it.

    Takes a base pipe and creates a parallel pipe by duplicating it.

    pipe

    The base data pipe

    returns

    a io.github.mandar2812.dynaml.pipes.ParallelPipe object.

  23. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  24. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  25. val extractTimeSeries: ((Double, Double, Double) ⇒ Double) ⇒ DataPipe[Stream[String], Stream[(Double, Double)]]

    Permalink

    This pipe assumes its input to be of the form "YYYY,Day,Hour,Value"

    This pipe assumes its input to be of the form "YYYY,Day,Hour,Value"

    It takes as input a function (TFunc) which converts a Tuple3 into a single "timestamp" like value.

    The pipe processes its data source line by line and outputs a Tuple2 in the following format.

    (Timestamp,Value)

    Usage: DynaMLPipe.extractTimeSeries(TFunc)

  26. val extractTimeSeriesVec: ((Double, Double, Double) ⇒ Double) ⇒ DataPipe[Stream[String], Stream[(Double, DenseVector[Double])]]

    Permalink

    This pipe is exactly similar to DynaMLPipe.extractTimeSeries, with one key difference, it returns a Tuple2 like (Timestamp, FeatureVector), where FeatureVector is a Vector of values.

  27. val extractTrainingFeatures: (List[Int], Map[Int, String]) ⇒ DataPipe[Stream[String], Stream[String]]

    Permalink

    Extract a subset of columns from a Stream of comma separated String also replace any missing value strings with the empty string.

    Extract a subset of columns from a Stream of comma separated String also replace any missing value strings with the empty string.

    Usage: DynaMLPipe.extractTrainingFeatures(List(1,2,3), Map(1 -> "N.A.", 2 -> "NA", 3 -> "na"))

  28. val fileToStream: DataPipe[String, Stream[String]]

    Permalink

    Data pipe which takes a file name/path as a String and returns a Stream of String.

  29. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  30. val gaussianScaling: DataPipe[Stream[(DenseVector[Double], DenseVector[Double])], (Stream[(DenseVector[Double], DenseVector[Double])], (GaussianScaler, GaussianScaler))]

    Permalink

    Scale a data set which is stored as a Stream, return the scaled data as well as a GaussianScaler instance which can be used to reverse the scaled values to the original data.

  31. val gaussianScalingTrainTest: DataPipe[(Stream[(DenseVector[Double], DenseVector[Double])], Stream[(DenseVector[Double], DenseVector[Double])]), (Stream[(DenseVector[Double], DenseVector[Double])], Stream[(DenseVector[Double], DenseVector[Double])], (GaussianScaler, GaussianScaler))]

    Permalink

    Perform gaussian normalization on a data stream which is a Tuple2 of the form.

    Perform gaussian normalization on a data stream which is a Tuple2 of the form.

    (Stream(training data), Stream(test data))

  32. def genericReplicationEncoder[I](n: Int)(implicit tag: ClassTag[I]): Encoder[I, Array[I]]

    Permalink
  33. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  34. def gpTuning[T, I](startingState: Map[String, Double], globalOpt: String = "GS", grid: Int = 3, step: Double = 0.02, maxIt: Int = 20, policy: String = "GS", prior: Map[String, ContinuousRVWithDistr[Double, ContinuousDistr[Double]]] = Map())(implicit arg0: ClassTag[I]): BifurcationPipe[AbstractGPRegressionModel[T, I], AbstractGPRegressionModel[T, I], Map[String, Double]]

    Permalink
  35. val groupedHaarWaveletFilter: (Array[Int]) ⇒ GroupedHaarWaveletFilter

    Permalink
  36. val haarWaveletFilter: (Int) ⇒ HaarWaveletFilter

    Permalink

    Constructs a data pipe which performs discrete Haar wavelet transform on a (breeze) vector signal.

  37. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  38. def identityPipe[T]: DataPipe[T, T]

    Permalink

    A trivial identity data pipe

  39. val invGroupedHaarWaveletFilter: (Array[Int]) ⇒ InvGroupedHaarWaveletFilter

    Permalink
  40. val invHaarWaveletFilter: (Int) ⇒ InverseHaarWaveletFilter

    Permalink

    Constructs a data pipe which performs inverse discrete Haar wavelet transform on a (breeze) vector signal.

  41. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  42. val logger: Logger

    Permalink
  43. val minMaxScaling: DataPipe[Stream[(DenseVector[Double], DenseVector[Double])], (Stream[(DenseVector[Double], DenseVector[Double])], (MinMaxScaler, MinMaxScaler))]

    Permalink

    Scale a data set which is stored as a Stream, return the scaled data as well as a MinMaxScaler instance which can be used to reverse the scaled values to the original data.

  44. val minMaxScalingTrainTest: DataPipe[(Stream[(DenseVector[Double], DenseVector[Double])], Stream[(DenseVector[Double], DenseVector[Double])]), (Stream[(DenseVector[Double], DenseVector[Double])], Stream[(DenseVector[Double], DenseVector[Double])], (MinMaxScaler, MinMaxScaler))]

    Permalink

    Perform [0,1] scaling on a data stream which is a Tuple2 of the form.

    Perform [0,1] scaling on a data stream which is a Tuple2 of the form.

    (Stream(training data), Stream(test data))

  45. def modelTuning[M <: GloballyOptWithGrad](startingState: Map[String, Double], globalOpt: String = "GS", grid: Int = 3, step: Double = 0.02): BifurcationPipe[M, M, Map[String, Double]]

    Permalink
  46. val multivariateGaussianScaling: DataPipe[Stream[(DenseVector[Double], DenseVector[Double])], (Stream[(DenseVector[Double], DenseVector[Double])], (MVGaussianScaler, MVGaussianScaler))]

    Permalink

    Scale a data set which is stored as a Stream, return the scaled data as well as a MVGaussianScaler instance which can be used to reverse the scaled values to the original data.

  47. val multivariateGaussianScalingTrainTest: DataPipe[(Stream[(DenseVector[Double], DenseVector[Double])], Stream[(DenseVector[Double], DenseVector[Double])]), (Stream[(DenseVector[Double], DenseVector[Double])], Stream[(DenseVector[Double], DenseVector[Double])], (MVGaussianScaler, MVGaussianScaler))]

    Permalink

    Scale a data set which is stored as a Stream, return the scaled data as well as a MVGaussianScaler instance which can be used to reverse the scaled values to the original data.

  48. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  49. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  50. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  51. val numeric_range: MetaPipe21[Double, Double, Int, Seq[Double]]

    Permalink

    Generate a numeric range by dividing an interval into bins.

  52. val pcaFeatureScaling: DataPipe[Stream[DenseVector[Double]], (Stream[DenseVector[Double]], PCAScaler)]

    Permalink

    Transform a data set by performing PCA on its patterns.

  53. val pcaScaling: DataPipe[Stream[(DenseVector[Double], DenseVector[Double])], (Stream[(DenseVector[Double], DenseVector[Double])], (PCAScaler, MVGaussianScaler))]

    Permalink

    Transform a data set consisting of features and targets.

    Transform a data set consisting of features and targets. Perform PCA scaling of features and gaussian scaling of targets.

  54. def rdfToGLM(modelName: String, y: String, xs: Array[String])(implicit renjin: RenjinScriptEngine): DataPipe[String, ListVector]

    Permalink

    Create a linear model from a R data frame.

    Create a linear model from a R data frame.

    modelName

    The name of the variable to store model

    y

    The name of the target variable

    xs

    A list of names denoting input variables

    returns

    A DataPipe which takes as input data frame variable name and returns a ListVector containing linear model attributes. Also stores the model in the variable given by modelName in the ongoing R session.

  55. val removeMissingLines: StreamFilterPipe[String]

    Permalink

    From a Stream of String remove all records which contain missing values, this pipe should be applied after the application of DynaMLPipe.extractTrainingFeatures.

  56. val replace: (String, String) ⇒ StreamMapPipe[String, String]

    Permalink

    Data pipe to replace all occurrences of a regular expression or string in a Stream of String with with a specified replacement string.

  57. val replaceWhiteSpaces: StreamMapPipe[String, String]

    Permalink

    Data pipe to replace all white spaces in a Stream of String with the comma character.

  58. def sgpTuning[T, I](startingState: Map[String, Double], globalOpt: String = "GS", grid: Int = 3, step: Double = 0.02, maxIt: Int = 20, prior: Map[String, ContinuousRVWithDistr[Double, ContinuousDistr[Double]]] = Map())(implicit arg0: ClassTag[I]): BifurcationPipe[ESGPModel[T, I], ESGPModel[T, I], Map[String, Double]]

    Permalink
  59. val splitFeaturesAndTargets: StreamMapPipe[String, (DenseVector[Double], Double)]

    Permalink

    Take each line which is a comma separated string and extract all but the last element into a feature vector and leave the last element as the "target" value.

    Take each line which is a comma separated string and extract all but the last element into a feature vector and leave the last element as the "target" value.

    This pipe outputs data in a Stream of Tuple2 in the following form

    (Vector(features), value)

  60. def splitTrainingTest[P](num_training: Int, num_test: Int): BifurcationPipe[(Stream[P], Stream[P]), Stream[P], Stream[P]]

    Permalink

    Extract a subset of the data into a Tuple2 which can be used as a training, test combo for model learning and evaluation.

    Extract a subset of the data into a Tuple2 which can be used as a training, test combo for model learning and evaluation.

    Usage: DynaMLPipe.splitTrainingTest(num_training, num_test)

  61. val streamToFile: (String) ⇒ SideEffectPipe[Stream[String]]

    Permalink

    Writes a Stream of String to a file.

    Writes a Stream of String to a file.

    Usage: DynaMLPipe.streamToFile("abc.csv")

  62. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  63. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  64. def trainParametricModel[G, T, Q, R, S, M <: ParameterizedLearner[G, T, Q, R, S]](regParameter: Double, step: Double = 0.05, maxIt: Int = 50, mini: Double = 1.0): DataPipe[M, M]

    Permalink
  65. val trimLines: StreamMapPipe[String, String]

    Permalink

    Trim white spaces from each line in a Stream of String

  66. val valuesToFile: (String) ⇒ SideEffectPipe[Stream[Seq[AnyVal]]]

    Permalink

    Writes a Stream of AnyVal to a file.

    Writes a Stream of AnyVal to a file.

    Usage: DynaMLPipe.valuesToFile("abc.csv")

  67. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  68. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  69. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Deprecated Value Members

  1. val featuresGaussianStandardization: DataPipe[(Stream[(DenseVector[Double], Double)], Stream[(DenseVector[Double], Double)]), ((Stream[(DenseVector[Double], Double)], Stream[(DenseVector[Double], Double)]), (DenseVector[Double], DenseVector[Double]))]

    Permalink

    Perform gaussian normalization on a data stream which is a Tuple2 of the form.

    Perform gaussian normalization on a data stream which is a Tuple2 of the form.

    (Stream(training data), Stream(test data))

    Annotations
    @deprecated
    Deprecated
  2. val trainTestGaussianStandardization: DataPipe[(Stream[(DenseVector[Double], Double)], Stream[(DenseVector[Double], Double)]), ((Stream[(DenseVector[Double], Double)], Stream[(DenseVector[Double], Double)]), (DenseVector[Double], DenseVector[Double]))]

    Permalink

    Perform gaussian normalization on a data stream which is a Tuple2 of the form.

    Perform gaussian normalization on a data stream which is a Tuple2 of the form.

    (Stream(training data), Stream(test data))

    Annotations
    @deprecated
    Deprecated
  3. val trainTestGaussianStandardizationMO: DataPipe[(Stream[(DenseVector[Double], DenseVector[Double])], Stream[(DenseVector[Double], DenseVector[Double])]), ((Stream[(DenseVector[Double], DenseVector[Double])], Stream[(DenseVector[Double], DenseVector[Double])]), (DenseVector[Double], DenseVector[Double]))]

    Permalink

    Perform gaussian normalization on a data stream which is a Tuple2 of the form.

    Perform gaussian normalization on a data stream which is a Tuple2 of the form.

    (Stream(training data), Stream(test data))

    Annotations
    @deprecated
    Deprecated

Inherited from AnyRef

Inherited from Any

Ungrouped