com.krux.hyperion.activity

CopyActivity

case class CopyActivity extends PipelineActivity[Ec2Resource] with Product with Serializable

The activity that copies data from one data node to the other.

Source
CopyActivity.scala
Note

it seems that both input and output format needs to be in CsvDataFormat for this copy to work properly and it needs to be a specific variance of the CSV, for more information check the web page:

http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-object-copyactivity.html

From our experience it's really hard to export using TsvDataFormat, in both import and export especially for tasks involving RedshiftCopyActivity. A general rule of thumb is always use default CsvDataFormat for tasks involving both exporting to S3 and copy to redshift.

Linear Supertypes
Serializable, Serializable, Product, Equals, PipelineActivity[Ec2Resource], NamedPipelineObject, PipelineObject, Ordered[PipelineObject], Comparable[PipelineObject], AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. CopyActivity
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. PipelineActivity
  7. NamedPipelineObject
  8. PipelineObject
  9. Ordered
  10. Comparable
  11. AnyRef
  12. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Type Members

  1. type Self = CopyActivity

    Definition Classes
    CopyActivity → PipelineActivity → NamedPipelineObject → PipelineObject

Value Members

  1. final def !=(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  2. final def !=(arg0: Any): Boolean

    Definition Classes
    Any

  3. final def ##(): Int

    Definition Classes
    AnyRef → Any

  4. def <(that: PipelineObject): Boolean

    Definition Classes
    Ordered

  5. def <=(that: PipelineObject): Boolean

    Definition Classes
    Ordered

  6. final def ==(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  7. final def ==(arg0: Any): Boolean

    Definition Classes
    Any

  8. def >(that: PipelineObject): Boolean

    Definition Classes
    Ordered

  9. def >=(that: PipelineObject): Boolean

    Definition Classes
    Ordered

  10. val activityFields: ActivityFields[Ec2Resource]

    Definition Classes
    CopyActivity → PipelineActivity

  11. final def asInstanceOf[T0]: T0

    Definition Classes
    Any

  12. def attemptTimeout: Option[HDuration]

    Definition Classes
    PipelineActivity

  13. val baseFields: BaseFields

    Definition Classes
    CopyActivity → NamedPipelineObject

  14. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  15. def compare(that: PipelineObject): Int

    Definition Classes
    PipelineObject → Ordered

  16. def compareTo(that: PipelineObject): Int

    Definition Classes
    Ordered → Comparable

  17. def dependsOn: Seq[PipelineActivity[_]]

    Definition Classes
    PipelineActivity

  18. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  19. def failureAndRerunMode: Option[FailureAndRerunMode]

    Definition Classes
    PipelineActivity

  20. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  21. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any

  22. def groupedBy(group: String): Self

    Postfix the name field

    Postfix the name field

    Definition Classes
    NamedPipelineObject

  23. def id: PipelineObjectId

    Definition Classes
    NamedPipelineObject → PipelineObject

  24. def idGroupedBy(group: String): Self

    Have a grouping postfix in the id field

    Have a grouping postfix in the id field

    Definition Classes
    NamedPipelineObject
    Note

    Id naming is more restrictive, it is recommended to not changing the id unleass you have a good reason

  25. def idNamed(namePrefix: String): Self

    Id field will be prefixed with name

    Id field will be prefixed with name

    Definition Classes
    NamedPipelineObject
    Note

    Id naming is more restrictive, it is recommended to not changing the id unless you have a good reason

  26. val input: Copyable

  27. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any

  28. def lateAfterTimeout: Option[HDuration]

    Definition Classes
    PipelineActivity

  29. def maxActiveInstances: Option[HInt]

    Definition Classes
    PipelineActivity

  30. def maximumRetries: Option[HInt]

    Definition Classes
    PipelineActivity

  31. def name: Option[String]

    Name of the pipeline object, if not set, it will defaults to

    Name of the pipeline object, if not set, it will defaults to 

    Option(id)
    Definition Classes
    NamedPipelineObject

  32. def named(namePrefix: String): Self

    Give the object a name prefix

    Give the object a name prefix

    Definition Classes
    NamedPipelineObject

  33. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef

  34. final def notify(): Unit

    Definition Classes
    AnyRef

  35. final def notifyAll(): Unit

    Definition Classes
    AnyRef

  36. def objects: Iterable[PipelineObject]

    Definition Classes
    CopyActivity → PipelineActivity → PipelineObject

  37. def onFail(alarms: SnsAlarm*): Self

    Definition Classes
    PipelineActivity

  38. def onFailAlarms: Seq[SnsAlarm]

    Definition Classes
    PipelineActivity

  39. def onLateAction(alarms: SnsAlarm*): Self

    Definition Classes
    PipelineActivity

  40. def onLateActionAlarms: Seq[SnsAlarm]

    Definition Classes
    PipelineActivity

  41. def onSuccess(alarms: SnsAlarm*): Self

    Definition Classes
    PipelineActivity

  42. def onSuccessAlarms: Seq[SnsAlarm]

    Definition Classes
    PipelineActivity

  43. val output: Copyable

  44. def preconditions: Seq[Precondition]

    Definition Classes
    PipelineActivity

  45. def ref: AdpRef[AdpActivity]

    Definition Classes
    PipelineActivity → PipelineObject

  46. def retryDelay: Option[HDuration]

    Definition Classes
    PipelineActivity

  47. def runsOn: Resource[Ec2Resource]

    Definition Classes
    PipelineActivity

  48. implicit def seq2Option[A](anySeq: Seq[A]): Option[Seq[A]]

    Definition Classes
    PipelineObject

  49. def seqToOption[A, B](anySeq: Seq[A])(transform: (A) ⇒ B): Option[Seq[B]]

    Definition Classes
    PipelineObject

  50. lazy val serialize: AdpCopyActivity

    Definition Classes
    CopyActivity → PipelineActivity → PipelineObject

  51. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef

  52. implicit def uniquePipelineId2String(id: PipelineObjectId): String

    Definition Classes
    PipelineObject

  53. def updateActivityFields(fields: ActivityFields[Ec2Resource]): CopyActivity

    Definition Classes
    CopyActivity → PipelineActivity

  54. def updateBaseFields(fields: BaseFields): CopyActivity

    Definition Classes
    CopyActivity → NamedPipelineObject

  55. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  56. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  57. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  58. def whenMet(conditions: Precondition*): Self

    Definition Classes
    PipelineActivity

  59. def withAttemptTimeout(duration: HDuration): Self

    Definition Classes
    PipelineActivity

  60. def withFailureAndRerunMode(mode: FailureAndRerunMode): Self

    Definition Classes
    PipelineActivity

  61. def withLateAfterTimeout(duration: HDuration): Self

    Definition Classes
    PipelineActivity

  62. def withMaxActiveInstances(activeInstances: HInt): Self

    Definition Classes
    PipelineActivity

  63. def withMaximumRetries(retries: HInt): Self

    Definition Classes
    PipelineActivity

  64. def withRetryDelay(duration: HDuration): Self

    Definition Classes
    PipelineActivity

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from PipelineActivity[Ec2Resource]

Inherited from NamedPipelineObject

Inherited from PipelineObject

Inherited from Ordered[PipelineObject]

Inherited from Comparable[PipelineObject]

Inherited from AnyRef

Inherited from Any

Ungrouped