Trait/Object

com.twitter.scalding

Config

Related Docs: object Config | package scalding

Permalink

trait Config extends Serializable

This is a wrapper class on top of Map[String, String]

Linear Supertypes
Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Config
  2. Serializable
  3. Serializable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract def toMap: Map[String, String]

    Permalink

Concrete Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. def +(kv: (String, String)): Config

    Permalink
  4. def ++(that: Config): Config

    Permalink
  5. def -(k: String): Config

    Permalink
  6. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  7. def addCascadingClassSerializationTokens(clazzes: Set[Class[_]]): Config

    Permalink

    The serialization of your data will be smaller if any classes passed between tasks in your job are listed here.

    The serialization of your data will be smaller if any classes passed between tasks in your job are listed here. Without this, strings are used to write the types IN EACH RECORD, which compression probably takes care of, but compression acts AFTER the data is serialized into buffers and spilling has been triggered.

  8. def addDistributedCacheFiles(cachedFiles: CachedFile*): Config

    Permalink

    Add files to be localized to the config.

    Add files to be localized to the config. Intended to be used by user code.

    cachedFiles

    CachedFiles to be added

    returns

    new Config with cached files

  9. def addFlowListener(flowListenerProvider: (Mode, Config) ⇒ FlowListener): Config

    Permalink

    configure flow listeneres for observability

  10. def addFlowStepListener(flowListenerProvider: (Mode, Config) ⇒ FlowStepListener): Config

    Permalink
  11. def addFlowStepStrategy(flowStrategyProvider: (Mode, Config) ⇒ FlowStepStrategy[JobConf]): Config

    Permalink
  12. def addReducerEstimator(clsName: String): Config

    Permalink

    Prepend an estimator so it will be tried first.

    Prepend an estimator so it will be tried first. If it returns None, the previously-set estimators will be tried in order.

  13. def addReducerEstimator[T](cls: Class[T]): Config

    Permalink

    Prepend an estimator so it will be tried first.

    Prepend an estimator so it will be tried first. If it returns None, the previously-set estimators will be tried in order.

  14. def addUniqueId(u: UniqueID): Config

    Permalink
  15. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  16. def clearFlowStepStrategies: Config

    Permalink
  17. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  18. def ensureUniqueId: (UniqueID, Config)

    Permalink

    Allocate a new UniqueID if there is not one present

  19. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  20. def equals(that: Any): Boolean

    Permalink
    Definition Classes
    Config → AnyRef → Any
  21. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  22. def get(key: String): Option[String]

    Permalink
  23. def getArgs: Args

    Permalink
  24. def getCascadingAppJar: Option[Try[Class[_]]]

    Permalink

    Returns None if not set, otherwise reflection is used to create the Class.forName

  25. def getCascadingAppName: Option[String]

    Permalink

    This is a name that if present is passed to flow.setName, which should appear in the job tracker.

  26. def getCascadingSerializationTokens: Map[Int, String]

    Permalink
  27. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  28. def getDistributedCachedFiles: Seq[CachedFile]

    Permalink

    Get cached files from config

  29. def getFlowListeners: List[Try[(Mode, Config) ⇒ FlowListener]]

    Permalink
  30. def getFlowStepListeners: List[Try[(Mode, Config) ⇒ FlowStepListener]]

    Permalink
  31. def getFlowStepStrategies: List[Try[(Mode, Config) ⇒ FlowStepStrategy[JobConf]]]

    Permalink
  32. def getHashJoinAutoForceRight: Boolean

    Permalink
  33. def getKryo: Option[KryoInstantiator]

    Permalink
  34. def getKryoRegisteredClasses: Set[Class[_]]

    Permalink

    This function gets the set of classes that have been registered to Kryo.

    This function gets the set of classes that have been registered to Kryo. They may or may not be used in this job, but Cascading might want to be made aware that these classes exist

  35. def getNumReducers: Option[Int]

    Permalink

    Get the number of reducers (this is the parameter Hadoop will use)

  36. def getRequireOrderedSerialization: Boolean

    Permalink
  37. def getScaldingExecutionId: Option[String]

    Permalink
  38. def getScaldingVersion: Option[String]

    Permalink
  39. def getSubmittedTimestamp: Option[RichDate]

    Permalink
  40. def getUniqueIds: Set[UniqueID]

    Permalink
  41. def hashCode(): Int

    Permalink
    Definition Classes
    Config → AnyRef → Any
  42. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  43. def maybeSetSubmittedTimestamp(date: RichDate = RichDate.now): (Option[RichDate], Config)

    Permalink
  44. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  45. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  46. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  47. def setArgs(args: Args): Config

    Permalink
  48. def setCascadingAppId(id: String): Config

    Permalink
  49. def setCascadingAppJar(clazz: Class[_]): Config

    Permalink

    Non-fat-jar use cases require this, BUT using it with fat jars can cause problems.

    Non-fat-jar use cases require this, BUT using it with fat jars can cause problems. It is not set by default, but if you have problems you might need to set the Job class here Consider also setting this same class here: setScaldingFlowClass

  50. def setCascadingAppName(name: String): Config

    Permalink
  51. def setDefaultComparator(clazz: Class[_ <: Comparator[_]]): Config

    Permalink
  52. def setHRavenHistoryUserName: Config

    Permalink

    Set username from System.used for querying hRaven.

  53. def setHashJoinAutoForceRight(b: Boolean): Config

    Permalink
  54. def setListSpillThreshold(count: Int): Config

    Permalink
  55. def setMapSideAggregationThreshold(count: Int): Config

    Permalink
  56. def setMapSpillThreshold(count: Int): Config

    Permalink
  57. def setNumReducers(n: Int): Config

    Permalink
  58. def setReducerEstimators(clsList: String): Config

    Permalink

    Set the entire list of reducer estimators (overriding the existing list)

  59. def setRequireOrderedSerialization(b: Boolean): Config

    Permalink

    Set this configuration option to require all grouping/cogrouping to use OrderedSerialization

  60. def setScaldingExecutionId(id: String): Config

    Permalink

    Set an ID to be shared across this usage of run for Execution

  61. def setScaldingFlowClass(clazz: Class[_]): Config

    Permalink
  62. def setScaldingVersion: Config

    Permalink
  63. def setSerialization(kryo: Either[(Class[_ <: KryoInstantiator], KryoInstantiator), Class[_ <: KryoInstantiator]], userHadoop: Seq[Class[_ <: Serialization[_]]] = Nil): Config

    Permalink
  64. def setVerboseFileSourceLogging(b: Boolean): Config

    Permalink

    Set to true to enable very verbose logging during FileSource's validation and planning.

    Set to true to enable very verbose logging during FileSource's validation and planning. This can help record what files were present / missing at runtime. Should only be enabled for debugging.

  65. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  66. def toString(): String

    Permalink
    Definition Classes
    AnyRef → Any
  67. def update[R](k: String)(fn: (Option[String]) ⇒ (Option[String], R)): (R, Config)

    Permalink
  68. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  69. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  70. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped