com.krux.hyperion.aws

AdpEmrActivity

case class AdpEmrActivity(id: String, name: Option[String], step: Seq[String], preStepCommand: Option[Seq[String]], postStepCommand: Option[Seq[String]], input: Option[Seq[AdpRef[AdpDataNode]]], output: Option[Seq[AdpRef[AdpDataNode]]], workerGroup: Option[String], runsOn: Option[AdpRef[AdpEmrCluster]], dependsOn: Option[Seq[AdpRef[AdpActivity]]], precondition: Option[Seq[AdpRef[AdpPrecondition]]], onFail: Option[Seq[AdpRef[AdpSnsAlarm]]], onSuccess: Option[Seq[AdpRef[AdpSnsAlarm]]], onLateAction: Option[Seq[AdpRef[AdpSnsAlarm]]], attemptTimeout: Option[String], lateAfterTimeout: Option[String], maximumRetries: Option[String], retryDelay: Option[String], failureAndRerunMode: Option[String], actionOnResourceFailure: Option[String], actionOnTaskFailure: Option[String]) extends AdpDataPipelineAbstractObject with AdpActivity with Product with Serializable

Runs an Amazon EMR job.

AWS Data Pipeline uses a different format for steps than Amazon EMR, for example AWS Data Pipeline uses comma-separated arguments after the JAR name in the EmrActivity step field.

step

One or more steps for the cluster to run. To specify multiple steps, up to 255, add multiple step fields. Use comma-separated arguments after the JAR name; for example, "s3://example-bucket/MyWork.jar,arg1,arg2,arg3".

preStepCommand

Shell scripts to be run before any steps are run. To specify multiple scripts, up to 255, add multiple preStepCommand fields.

postStepCommand

Shell scripts to be run after all steps are finished. To specify multiple scripts, up to 255, add multiple postStepCommand fields.

input

The input data source.

output

The location for the output

runsOn

The Amazon EMR cluster to run this cluster.

actionOnResourceFailure

Action for the EmrCluster to take when it fails. String: retryall (retry all inputs) or retrynone (retry nothing)

actionOnTaskFailure

Action for the activity/task to take when its associated EmrCluster fails. String: continue (do not terminate the cluster) or terminate

Source
AdpActivities.scala
Linear Supertypes
Serializable, Serializable, Product, Equals, AdpActivity, AdpDataPipelineObject, AdpDataPipelineAbstractObject, AdpObject, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. AdpEmrActivity
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. AdpActivity
  7. AdpDataPipelineObject
  8. AdpDataPipelineAbstractObject
  9. AdpObject
  10. AnyRef
  11. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new AdpEmrActivity(id: String, name: Option[String], step: Seq[String], preStepCommand: Option[Seq[String]], postStepCommand: Option[Seq[String]], input: Option[Seq[AdpRef[AdpDataNode]]], output: Option[Seq[AdpRef[AdpDataNode]]], workerGroup: Option[String], runsOn: Option[AdpRef[AdpEmrCluster]], dependsOn: Option[Seq[AdpRef[AdpActivity]]], precondition: Option[Seq[AdpRef[AdpPrecondition]]], onFail: Option[Seq[AdpRef[AdpSnsAlarm]]], onSuccess: Option[Seq[AdpRef[AdpSnsAlarm]]], onLateAction: Option[Seq[AdpRef[AdpSnsAlarm]]], attemptTimeout: Option[String], lateAfterTimeout: Option[String], maximumRetries: Option[String], retryDelay: Option[String], failureAndRerunMode: Option[String], actionOnResourceFailure: Option[String], actionOnTaskFailure: Option[String])

    step

    One or more steps for the cluster to run. To specify multiple steps, up to 255, add multiple step fields. Use comma-separated arguments after the JAR name; for example, "s3://example-bucket/MyWork.jar,arg1,arg2,arg3".

    preStepCommand

    Shell scripts to be run before any steps are run. To specify multiple scripts, up to 255, add multiple preStepCommand fields.

    postStepCommand

    Shell scripts to be run after all steps are finished. To specify multiple scripts, up to 255, add multiple postStepCommand fields.

    input

    The input data source.

    output

    The location for the output

    runsOn

    The Amazon EMR cluster to run this cluster.

    actionOnResourceFailure

    Action for the EmrCluster to take when it fails. String: retryall (retry all inputs) or retrynone (retry nothing)

    actionOnTaskFailure

    Action for the activity/task to take when its associated EmrCluster fails. String: continue (do not terminate the cluster) or terminate

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any

  3. final def ##(): Int

    Definition Classes
    AnyRef → Any

  4. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  5. final def ==(arg0: Any): Boolean

    Definition Classes
    Any

  6. val actionOnResourceFailure: Option[String]

    Action for the EmrCluster to take when it fails.

    Action for the EmrCluster to take when it fails. String: retryall (retry all inputs) or retrynone (retry nothing)

  7. val actionOnTaskFailure: Option[String]

    Action for the activity/task to take when its associated EmrCluster fails.

    Action for the activity/task to take when its associated EmrCluster fails. String: continue (do not terminate the cluster) or terminate

  8. final def asInstanceOf[T0]: T0

    Definition Classes
    Any

  9. val attemptTimeout: Option[String]

    The timeout time interval for an object attempt.

    The timeout time interval for an object attempt. If an attempt does not complete within the start time plus this time interval, AWS Data Pipeline marks the attempt as failed and your retry settings determine the next steps taken.

    Definition Classes
    AdpEmrActivity → AdpActivity

  10. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  11. val dependsOn: Option[Seq[AdpRef[AdpActivity]]]

    One or more references to other Activities that must reach the FINISHED state before this activity will start.

    One or more references to other Activities that must reach the FINISHED state before this activity will start.

    Definition Classes
    AdpEmrActivity → AdpActivity

  12. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  13. val failureAndRerunMode: Option[String]

    Determines whether pipeline object failures and rerun commands cascade through pipeline object dependencies

    Determines whether pipeline object failures and rerun commands cascade through pipeline object dependencies

    Possible values include cascade and none.

    Definition Classes
    AdpEmrActivity → AdpActivity

  14. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  15. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any

  16. val id: String

    The ID of the object, IDs must be unique within a pipeline definition

    The ID of the object, IDs must be unique within a pipeline definition

    Definition Classes
    AdpEmrActivity → AdpDataPipelineObject → AdpObject

  17. val input: Option[Seq[AdpRef[AdpDataNode]]]

    The input data source.

  18. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any

  19. val lateAfterTimeout: Option[String]

    The time period in which the object run must start.

    The time period in which the object run must start. If the object does not start within the scheduled start time plus this time interval, it is considered late

    Definition Classes
    AdpEmrActivity → AdpActivity

  20. val maximumRetries: Option[String]

    The maximum number of times to retry the action.

    The maximum number of times to retry the action. The default value is 2, which results in 3 tries total (1 original attempt plus 2 retries). The maximum value is 5 (6 total attempts).

    Definition Classes
    AdpEmrActivity → AdpActivity

  21. val name: Option[String]

    The optional, user-defined label of the object.

    The optional, user-defined label of the object. If you do not provide a name for an object in a pipeline definition, AWS Data Pipeline automatically duplicates the value of id.

    Definition Classes
    AdpEmrActivity → AdpDataPipelineObject → AdpDataPipelineAbstractObject

  22. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  23. final def notify(): Unit

    Definition Classes
    AnyRef

  24. final def notifyAll(): Unit

    Definition Classes
    AnyRef

  25. val onFail: Option[Seq[AdpRef[AdpSnsAlarm]]]

    The SNS alarm to raise when the activity fails.

    The SNS alarm to raise when the activity fails.

    Definition Classes
    AdpEmrActivity → AdpActivity

  26. val onLateAction: Option[Seq[AdpRef[AdpSnsAlarm]]]

    The SNS alarm to raise when the activity fails to start on time.

    The SNS alarm to raise when the activity fails to start on time.

    Definition Classes
    AdpEmrActivity → AdpActivity

  27. val onSuccess: Option[Seq[AdpRef[AdpSnsAlarm]]]

    The SNS alarm to raise when the activity succeeds.

    The SNS alarm to raise when the activity succeeds.

    Definition Classes
    AdpEmrActivity → AdpActivity

  28. val output: Option[Seq[AdpRef[AdpDataNode]]]

    The location for the output

  29. val postStepCommand: Option[Seq[String]]

    Shell scripts to be run after all steps are finished.

    Shell scripts to be run after all steps are finished. To specify multiple scripts, up to 255, add multiple postStepCommand fields.

  30. val preStepCommand: Option[Seq[String]]

    Shell scripts to be run before any steps are run.

    Shell scripts to be run before any steps are run. To specify multiple scripts, up to 255, add multiple preStepCommand fields.

  31. val precondition: Option[Seq[AdpRef[AdpPrecondition]]]

    A condition that must be met before the object can run.

    A condition that must be met before the object can run. To specify multiple conditions, add multiple precondition fields. The activity cannot run until all its conditions are met.

    Definition Classes
    AdpEmrActivity → AdpActivity

  32. val retryDelay: Option[String]

    The timeout duration between two retry attempts.

    The timeout duration between two retry attempts. The default is 10 minutes.

    Definition Classes
    AdpEmrActivity → AdpActivity

  33. val runsOn: Option[AdpRef[AdpEmrCluster]]

    The Amazon EMR cluster to run this cluster.

  34. val step: Seq[String]

    One or more steps for the cluster to run.

    One or more steps for the cluster to run. To specify multiple steps, up to 255, add multiple step fields. Use comma-separated arguments after the JAR name; for example, "s3://example-bucket/MyWork.jar,arg1,arg2,arg3".

  35. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef

  36. val type: String

    The type of object.

    The type of object. Use one of the predefined AWS Data Pipeline object types.

    Definition Classes
    AdpEmrActivity → AdpDataPipelineObject

  37. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  38. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  39. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  40. val workerGroup: Option[String]

    The worker group.

    The worker group. This is used for routing tasks. If you provide a runsOn value and workerGroup exists, workerGroup is ignored.

    Definition Classes
    AdpEmrActivity → AdpActivity

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from AdpActivity

Inherited from AdpDataPipelineObject

Inherited from AdpDataPipelineAbstractObject

Inherited from AdpObject

Inherited from AnyRef

Inherited from Any

Ungrouped