Class

org.apache.spark.ml.odkl.hyperopt

GroupedSearch

Related Doc: package hyperopt

Permalink

class GroupedSearch[ModelIn <: ModelWithSummary[ModelIn]] extends ForkedEstimator[ModelIn, OptimizerStage, ModelIn] with HasConfigurations

Utility used to perform stepwise search for hyper-parameters. Usefull in case if there are certain groups of parameters which do not influence each other and can be optimized separatelly. Pass sequence of optimizers to the grouped search to apply sequential optimization.

NB: Grouped search must itself be configured to use single thread, but nested optimizers allowed to use as many threads as they need.

Linear Supertypes
HasConfigurations, ForkedEstimator[ModelIn, OptimizerStage, ModelIn], ForkedModelParams, SummarizableEstimator[ModelIn], Estimator[ModelIn], PipelineStage, Logging, Params, Serializable, Serializable, Identifiable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. GroupedSearch
  2. HasConfigurations
  3. ForkedEstimator
  4. ForkedModelParams
  5. SummarizableEstimator
  6. Estimator
  7. PipelineStage
  8. Logging
  9. Params
  10. Serializable
  11. Serializable
  12. Identifiable
  13. AnyRef
  14. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new GroupedSearch(nested: Seq[(String, HyperparametersOptimizer[ModelIn])])

    Permalink

  2. new GroupedSearch(nested: Seq[(String, HyperparametersOptimizer[ModelIn])], uid: String)

    Permalink

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def $[T](param: Param[T]): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  4. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  5. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  6. final def clear(param: Param[_]): GroupedSearch.this.type

    Permalink
    Definition Classes
    Params

  7. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  8. val configurations: Block

    Permalink
    Definition Classes
    HasConfigurations

  9. def copy(extra: ParamMap): GroupedSearch[ModelIn]

    Permalink
    Definition Classes
    GroupedSearch → SummarizableEstimator → Estimator → PipelineStage → Params

  10. def copyValues[T <: Params](to: T, extra: ParamMap): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  11. def createForkSource(dataset: Dataset[_]): ForkSource[ModelIn, OptimizerStage, ModelIn]

    Permalink
    Attributes
    protected
    Definition Classes
    GroupedSearch → ForkedEstimator

  12. def createForks(dataset: Dataset[_]): Seq[(OptimizerStage, DataFrame)]

    Permalink

    Override this method and create forks to train from the data.

    Override this method and create forks to train from the data.

    Attributes
    protected
    Definition Classes
    GroupedSearch → ForkedEstimator

  13. final def defaultCopy[T <: Params](extra: ParamMap): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  14. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  15. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  16. def explainParam(param: Param[_]): String

    Permalink
    Definition Classes
    Params

  17. def explainParams(): String

    Permalink
    Definition Classes
    Params

  18. final def extractParamMap(): ParamMap

    Permalink
    Definition Classes
    Params

  19. final def extractParamMap(extra: ParamMap): ParamMap

    Permalink
    Definition Classes
    Params

  20. def failFast(key: OptimizerStage, triedIn: Try[ModelIn]): Try[ModelIn]

    Permalink
    Attributes
    protected
    Definition Classes
    ForkedEstimator

  21. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  22. def fit(dataset: Dataset[_]): ModelIn

    Permalink
    Definition Classes
    GroupedSearch → ForkedEstimator → Estimator

  23. def fit(dataset: Dataset[_], paramMaps: Array[ParamMap]): Seq[ModelIn]

    Permalink
    Definition Classes
    Estimator
    Annotations
    @Since( "2.0.0" )

  24. def fit(dataset: Dataset[_], paramMap: ParamMap): ModelIn

    Permalink
    Definition Classes
    Estimator
    Annotations
    @Since( "2.0.0" )

  25. def fit(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): ModelIn

    Permalink
    Definition Classes
    Estimator
    Annotations
    @Since( "2.0.0" ) @varargs()

  26. def fitFork(estimator: SummarizableEstimator[ModelIn], wholeData: Dataset[_], partialData: (OptimizerStage, DataFrame)): (OptimizerStage, Try[ModelIn])

    Permalink
    Definition Classes
    GroupedSearch → ForkedEstimator

  27. final def get[T](param: Param[T]): Option[T]

    Permalink
    Definition Classes
    Params

  28. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  29. def getCurrentContext: Seq[String]

    Permalink
    Attributes
    protected
    Definition Classes
    ForkedEstimator

  30. final def getDefault[T](param: Param[T]): Option[T]

    Permalink
    Definition Classes
    Params

  31. final def getOrDefault[T](param: Param[T]): T

    Permalink
    Definition Classes
    Params

  32. def getParam(paramName: String): Param[Any]

    Permalink
    Definition Classes
    Params

  33. final def hasDefault[T](param: Param[T]): Boolean

    Permalink
    Definition Classes
    Params

  34. def hasParam(paramName: String): Boolean

    Permalink
    Definition Classes
    Params

  35. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  36. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  37. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  38. final def isDefined(param: Param[_]): Boolean

    Permalink
    Definition Classes
    Params

  39. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  40. final def isSet(param: Param[_]): Boolean

    Permalink
    Definition Classes
    Params

  41. def isTraceEnabled(): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  42. def log: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  43. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  44. def logDebug(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  45. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  46. def logError(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  47. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  48. def logInfo(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  49. def logName: String

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  50. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  51. def logTrace(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  52. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  53. def logWarning(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  54. def mayBePropagateKey(data: DataFrame, key: Any): DataFrame

    Permalink
    Attributes
    protected
    Definition Classes
    ForkedModelParams

  55. def mergeModels(sqlContext: SQLContext, models: Seq[(OptimizerStage, Try[ModelIn])]): ModelIn

    Permalink

    Given models trained for each fork create a combined model.

    Given models trained for each fork create a combined model. This model is the result of the estimator.

    Attributes
    protected
    Definition Classes
    GroupedSearch → ForkedEstimator

  56. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  57. val nested: SummarizableEstimator[ModelIn]

    Permalink

    Nested estimator to call for each fork.

    Nested estimator to call for each fork.

    Definition Classes
    ForkedEstimator

  58. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  59. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  60. final val numThreads: IntParam

    Permalink
    Definition Classes
    ForkedEstimator

  61. final val overwriteModels: BooleanParam

    Permalink
    Definition Classes
    ForkedEstimator

  62. lazy val params: Array[Param[_]]

    Permalink
    Definition Classes
    Params

  63. final val pathForTempModels: Param[String]

    Permalink
    Definition Classes
    ForkedEstimator

  64. final val persistingKeyColumns: StringArrayParam

    Permalink
    Definition Classes
    ForkedEstimator

  65. final val propagatedKeyColumn: Param[String]

    Permalink
    Definition Classes
    ForkedModelParams

  66. final def set(paramPair: ParamPair[_]): GroupedSearch.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  67. final def set(param: String, value: Any): GroupedSearch.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  68. final def set[T](param: Param[T], value: T): GroupedSearch.this.type

    Permalink
    Definition Classes
    Params

  69. final def setDefault(paramPairs: ParamPair[_]*): GroupedSearch.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  70. final def setDefault[T](param: Param[T], value: T): GroupedSearch.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  71. def setNumThreads(value: Int): GroupedSearch.this.type

    Permalink
    Definition Classes
    ForkedEstimator

  72. def setOverwriteModels(value: Boolean): GroupedSearch.this.type

    Permalink
    Definition Classes
    ForkedEstimator

  73. def setPathForTempModels(value: String): GroupedSearch.this.type

    Permalink
    Definition Classes
    ForkedEstimator

  74. def setPropagatedKeyColumn(value: String): GroupedSearch.this.type

    Permalink
    Definition Classes
    ForkedModelParams

  75. val stageNameColumnName: Param[String]

    Permalink

  76. val stageReversedIndexColumnName: Param[String]

    Permalink

  77. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  78. def toString(): String

    Permalink
    Definition Classes
    Identifiable → AnyRef → Any

  79. def transformSchema(schema: StructType): StructType

    Permalink
    Definition Classes
    ForkedEstimator → PipelineStage
    Annotations
    @DeveloperApi()

  80. def transformSchema(schema: StructType, logging: Boolean): StructType

    Permalink
    Attributes
    protected
    Definition Classes
    PipelineStage
    Annotations
    @DeveloperApi()

  81. val uid: String

    Permalink
    Definition Classes
    GroupedSearch → ForkedEstimator → Identifiable

  82. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  83. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  84. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from HasConfigurations

Inherited from ForkedEstimator[ModelIn, OptimizerStage, ModelIn]

Inherited from ForkedModelParams

Inherited from SummarizableEstimator[ModelIn]

Inherited from Estimator[ModelIn]

Inherited from PipelineStage

Inherited from Logging

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

Ungrouped