com.krux.hyperion.activity

HiveActivity

case class HiveActivity[A <: EmrCluster] extends EmrTaskActivity[A] with Product with Serializable

Runs a Hive query on an Amazon EMR cluster. HiveActivity makes it easier to set up an Amzon EMR activity and automatically creates Hive tables based on input data coming in from either Amazon S3 or Amazon RDS. All you need to specify is the HiveQL to run on the source data. AWS Data Pipeline automatically creates Hive tables with ${input1}, ${input2}, etc. based on the input fields in the Hive Activity object. For S3 inputs, the dataFormat field is used to create the Hive column names. For MySQL (RDS) inputs, the column names for the SQL query are used to create the Hive column names.

Source
HiveActivity.scala
Linear Supertypes
Serializable, Serializable, Product, Equals, EmrTaskActivity[A], EmrActivity[A], PipelineActivity[A], NamedPipelineObject, PipelineObject, Ordered[PipelineObject], Comparable[PipelineObject], AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. HiveActivity
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. EmrTaskActivity
  7. EmrActivity
  8. PipelineActivity
  9. NamedPipelineObject
  10. PipelineObject
  11. Ordered
  12. Comparable
  13. AnyRef
  14. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Type Members

  1. type Self = HiveActivity[A]

    Definition Classes
    HiveActivity → EmrTaskActivity → EmrActivity → PipelineActivity → NamedPipelineObject → PipelineObject

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any

  3. final def ##(): Int

    Definition Classes
    AnyRef → Any

  4. def <(that: PipelineObject): Boolean

    Definition Classes
    Ordered

  5. def <=(that: PipelineObject): Boolean

    Definition Classes
    Ordered

  6. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  7. final def ==(arg0: Any): Boolean

    Definition Classes
    Any

  8. def >(that: PipelineObject): Boolean

    Definition Classes
    Ordered

  9. def >=(that: PipelineObject): Boolean

    Definition Classes
    Ordered

  10. val activityFields: ActivityFields[A]

    Definition Classes
    HiveActivity → PipelineActivity

  11. final def asInstanceOf[T0]: T0

    Definition Classes
    Any

  12. def attemptTimeout: Option[HDuration]

    Definition Classes
    PipelineActivity

  13. val baseFields: BaseFields

    Definition Classes
    HiveActivity → NamedPipelineObject

  14. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  15. def compare(that: PipelineObject): Int

    Definition Classes
    PipelineObject → Ordered

  16. def compareTo(that: PipelineObject): Int

    Definition Classes
    Ordered → Comparable

  17. def dependsOn: Seq[PipelineActivity[_]]

    Definition Classes
    PipelineActivity

  18. val emrTaskActivityFields: EmrTaskActivityFields

    Definition Classes
    HiveActivity → EmrTaskActivity

  19. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  20. def failureAndRerunMode: Option[FailureAndRerunMode]

    Definition Classes
    PipelineActivity

  21. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  22. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any

  23. def groupedBy(group: String): Self

    Postfix the name field

    Postfix the name field

    Definition Classes
    NamedPipelineObject

  24. val hadoopQueue: Option[HString]

  25. val hiveScript: Script

  26. def id: PipelineObjectId

    Definition Classes
    NamedPipelineObject → PipelineObject

  27. def idGroupedBy(group: String): Self

    Have a grouping postfix in the id field

    Have a grouping postfix in the id field

    Definition Classes
    NamedPipelineObject
    Note

    Id naming is more restrictive, it is recommended to not changing the id unleass you have a good reason

  28. def idNamed(namePrefix: String): Self

    Id field will be prefixed with name

    Id field will be prefixed with name

    Definition Classes
    NamedPipelineObject
    Note

    Id naming is more restrictive, it is recommended to not changing the id unless you have a good reason

  29. val input: Seq[DataNode]

  30. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any

  31. def lateAfterTimeout: Option[HDuration]

    Definition Classes
    PipelineActivity

  32. def maxActiveInstances: Option[HInt]

    Definition Classes
    PipelineActivity

  33. def maximumRetries: Option[HInt]

    Definition Classes
    PipelineActivity

  34. def name: Option[String]

    Name of the pipeline object, if not set, it will defaults to

    Name of the pipeline object, if not set, it will defaults to 

    Option(id)
    Definition Classes
    NamedPipelineObject

  35. def named(namePrefix: String): Self

    Give the object a name prefix

    Give the object a name prefix

    Definition Classes
    NamedPipelineObject

  36. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  37. final def notify(): Unit

    Definition Classes
    AnyRef

  38. final def notifyAll(): Unit

    Definition Classes
    AnyRef

  39. def objects: Iterable[PipelineObject]

    Definition Classes
    HiveActivity → EmrTaskActivity → PipelineActivity → PipelineObject

  40. def onFail(alarms: SnsAlarm*): Self

    Definition Classes
    PipelineActivity

  41. def onFailAlarms: Seq[SnsAlarm]

    Definition Classes
    PipelineActivity

  42. def onLateAction(alarms: SnsAlarm*): Self

    Definition Classes
    PipelineActivity

  43. def onLateActionAlarms: Seq[SnsAlarm]

    Definition Classes
    PipelineActivity

  44. def onSuccess(alarms: SnsAlarm*): Self

    Definition Classes
    PipelineActivity

  45. def onSuccessAlarms: Seq[SnsAlarm]

    Definition Classes
    PipelineActivity

  46. val output: Seq[DataNode]

  47. def postActivityTaskConfig: Option[ShellScriptConfig]

    Definition Classes
    EmrTaskActivity

  48. def preActivityTaskConfig: Option[ShellScriptConfig]

    Definition Classes
    EmrTaskActivity

  49. def preconditions: Seq[Precondition]

    Definition Classes
    PipelineActivity

  50. def ref: AdpRef[AdpActivity]

    Definition Classes
    PipelineActivity → PipelineObject

  51. def retryDelay: Option[HDuration]

    Definition Classes
    PipelineActivity

  52. def runsOn: Resource[A]

    Definition Classes
    PipelineActivity

  53. val scriptVariables: Seq[HString]

  54. implicit def seq2Option[A](anySeq: Seq[A]): Option[Seq[A]]

    Definition Classes
    PipelineObject

  55. def seqToOption[A, B](anySeq: Seq[A])(transform: (A) ⇒ B): Option[Seq[B]]

    Definition Classes
    PipelineObject

  56. lazy val serialize: AdpHiveActivity

    Definition Classes
    HiveActivity → PipelineActivity → PipelineObject

  57. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef

  58. implicit def uniquePipelineId2String(id: PipelineObjectId): String

    Definition Classes
    PipelineObject

  59. def updateActivityFields(fields: ActivityFields[A]): HiveActivity[A]

    Definition Classes
    HiveActivity → PipelineActivity

  60. def updateBaseFields(fields: BaseFields): HiveActivity[A]

    Definition Classes
    HiveActivity → NamedPipelineObject

  61. def updateEmrTaskActivityFields(fields: EmrTaskActivityFields): HiveActivity[A]

    Definition Classes
    HiveActivity → EmrTaskActivity

  62. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  63. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  64. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  65. def whenMet(conditions: Precondition*): Self

    Definition Classes
    PipelineActivity

  66. def withAttemptTimeout(duration: HDuration): Self

    Definition Classes
    PipelineActivity

  67. def withFailureAndRerunMode(mode: FailureAndRerunMode): Self

    Definition Classes
    PipelineActivity

  68. def withHadoopQueue(queue: HString): HiveActivity[A]

  69. def withLateAfterTimeout(duration: HDuration): Self

    Definition Classes
    PipelineActivity

  70. def withMaxActiveInstances(activeInstances: HInt): Self

    Definition Classes
    PipelineActivity

  71. def withMaximumRetries(retries: HInt): Self

    Definition Classes
    PipelineActivity

  72. def withPostActivityTaskConfig(config: ShellScriptConfig): Self

    Definition Classes
    EmrTaskActivity

  73. def withPreActivityTaskConfig(config: ShellScriptConfig): Self

    Definition Classes
    EmrTaskActivity

  74. def withRetryDelay(duration: HDuration): Self

    Definition Classes
    PipelineActivity

  75. def withScriptVariable(scriptVariable: HString*): HiveActivity[A]

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from EmrTaskActivity[A]

Inherited from EmrActivity[A]

Inherited from PipelineActivity[A]

Inherited from NamedPipelineObject

Inherited from PipelineObject

Inherited from Ordered[PipelineObject]

Inherited from Comparable[PipelineObject]

Inherited from AnyRef

Inherited from Any

Ungrouped