The input data source.
The location for the output
Shell scripts to be run before any steps are run. To specify multiple scripts, up to 255, add multiple preStepCommand fields.
Shell scripts to be run after all steps are finished. To specify multiple scripts, up to 255, add multiple postStepCommand fields.
Action for the EmrCluster to take when it fails. String: retryall (retry all inputs) or retrynone (retry nothing)
Action for the activity/task to take when its associated EmrCluster fails. String: continue (do not terminate the cluster) or terminate
One or more steps for the cluster to run. To specify multiple steps, up to 255, add multiple step fields. Use comma-separated arguments after the JAR name; for example, "s3://example-bucket/MyWork.jar,arg1,arg2,arg3".
The Amazon EMR cluster to run this cluster.
Action for the EmrCluster to take when it fails.
Action for the EmrCluster to take when it fails. String: retryall (retry all inputs) or retrynone (retry nothing)
Action for the activity/task to take when its associated EmrCluster fails.
Action for the activity/task to take when its associated EmrCluster fails. String: continue (do not terminate the cluster) or terminate
One or more references to other Activities that must reach the FINISHED state before this activity will start.
One or more references to other Activities that must reach the FINISHED state before this activity will start.
The ID of the object, IDs must be unique within a pipeline definition
The ID of the object, IDs must be unique within a pipeline definition
The input data source.
The optional, user-defined label of the object.
The optional, user-defined label of the object. If you do not provide a name for an object in a pipeline definition, AWS Data Pipeline automatically duplicates the value of id.
The SNS alarm to raise when the activity fails.
The SNS alarm to raise when the activity fails.
The SNS alarm to raise when the activity fails to start on time.
The SNS alarm to raise when the activity fails to start on time.
The SNS alarm to raise when the activity succeeds.
The SNS alarm to raise when the activity succeeds.
The location for the output
Shell scripts to be run after all steps are finished.
Shell scripts to be run after all steps are finished. To specify multiple scripts, up to 255, add multiple postStepCommand fields.
Shell scripts to be run before any steps are run.
Shell scripts to be run before any steps are run. To specify multiple scripts, up to 255, add multiple preStepCommand fields.
A condition that must be met before the object can run.
A condition that must be met before the object can run. To specify multiple conditions, add multiple precondition fields. The activity cannot run until all its conditions are met.
The Amazon EMR cluster to run this cluster.
The Amazon EMR cluster to run this cluster.
One or more steps for the cluster to run.
One or more steps for the cluster to run. To specify multiple steps, up to 255, add multiple step fields. Use comma-separated arguments after the JAR name; for example, "s3://example-bucket/MyWork.jar,arg1,arg2,arg3".
The type of object.
The type of object. Use one of the predefined AWS Data Pipeline object types.
Runs an Amazon EMR cluster.
AWS Data Pipeline uses a different format for steps than Amazon EMR, for example AWS Data Pipeline uses comma-separated arguments after the JAR name in the EmrActivity step field.
The input data source.
The location for the output
Shell scripts to be run before any steps are run. To specify multiple scripts, up to 255, add multiple preStepCommand fields.
Shell scripts to be run after all steps are finished. To specify multiple scripts, up to 255, add multiple postStepCommand fields.
Action for the EmrCluster to take when it fails. String: retryall (retry all inputs) or retrynone (retry nothing)
Action for the activity/task to take when its associated EmrCluster fails. String: continue (do not terminate the cluster) or terminate
One or more steps for the cluster to run. To specify multiple steps, up to 255, add multiple step fields. Use comma-separated arguments after the JAR name; for example, "s3://example-bucket/MyWork.jar,arg1,arg2,arg3".
The Amazon EMR cluster to run this cluster.