objects

Type Members

1. case class CopyActivity extends PipelineActivity with Product with Serializable

   

   The activity that copys data from one data node to the other.

   The activity that copys data from one data node to the other.

   Note

   it seems that both input and output format needs to be in CsvDataFormat for this copy to work properly and it needs to be a specific variance of the CSV, for more information check the web page:

   http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-object-copyactivity.html

   From our experience it's really hard to export using TsvDataFormat, in both import and export especially for tasks involving RedshiftCopyActivity. A general rule of thumb is always use default CsvDataFormat for tasks involving both exporting to S3 and copy to redshift.

2. trait Copyable extends DataNode

   

3. case class CsvDataFormat extends DataFormat with Product with Serializable

   

   CSV data format

4. case class CustomDataFormat extends DataFormat with Product with Serializable

   

   Custom data format

5. trait DataFormat extends PipelineObject

   

6. trait DataNode extends PipelineObject

   

7. trait Database extends PipelineObject

   

8. case class DefaultObject(schedule: Schedule)(implicit hc: HyperionContext) extends PipelineObject with Product with Serializable

   

9. case class DeleteS3PathActivity extends PipelineActivity with Product with Serializable

   

   Activity to recursively delete files in an S3 path.

10. case class DoubleParameter(id: String, value: Double, description: Option[String] = None, allowedValues: Seq[Double] = Seq(), encrypted: Boolean = false) extends Parameter with Product with Serializable

    

11. case class DynamoDBDataExistsPrecondition extends Precondition with Product with Serializable

    

    A precondition to check that data exists in a DynamoDB table.

12. case class DynamoDBDataFormat extends DataFormat with Product with Serializable

    

    DynamoDB data format

13. case class DynamoDBExportDataFormat extends DataFormat with Product with Serializable

    

    DynamoDB Export data format

14. case class DynamoDBTableExistsPrecondition extends Precondition with Product with Serializable

    

    A precondition to check that the DynamoDB table exists.

15. case class Ec2Resource extends ResourceObject with Product with Serializable

    

    EC2 resource

16. trait EmrActivity extends PipelineActivity

    

17. trait EmrCluster extends ResourceObject

    

18. case class ExistsPrecondition extends Precondition with Product with Serializable

    

    Checks whether a data node object exists.

19. case class FixedObjectId(seed: String) extends PipelineObjectId with Product with Serializable

    

20. trait GoogleStorageActivity extends PipelineActivity

    

21. case class GoogleStorageDownloadActivity extends GoogleStorageActivity with Product with Serializable

    

    Google Storage Download activity

22. case class GoogleStorageUploadActivity extends GoogleStorageActivity with Product with Serializable

    

    Google Storage Upload activity

23. case class HiveActivity extends PipelineActivity with Product with Serializable

    

24. case class HiveCopyActivity extends PipelineActivity with Product with Serializable

    

25. case class IntegerParameter(id: String, value: Int, description: Option[String] = None, allowedValues: Seq[Int] = Seq(), encrypted: Boolean = false) extends Parameter with Product with Serializable

    

26. case class JarActivity extends PipelineActivity with Product with Serializable

    

    Shell command activity

27. trait JdbcDatabase extends Database

    

28. case class MapReduceActivity extends EmrActivity with Product with Serializable

    

    Defines a MapReduce activity

29. case class MapReduceCluster extends EmrCluster with Product with Serializable

    

    Launch a MapReduce cluster

30. case class MapReduceStep(jar: String = "", mainClass: String = "", args: Seq[String] = List()) extends Product with Serializable

    

    A MapReduce step that runs on MapReduce Cluster

31. case class NameGroupObjectId(name: String, group: String) extends PipelineObjectId with Product with Serializable

    

32. trait Parameter extends AnyRef

    

33. case class PigActivity extends PipelineActivity with Product with Serializable

    

34. trait PipelineActivity extends PipelineObject

    

35. trait PipelineObject extends AnyRef

    

    The base trait of krux data pipeline objects.

36. trait PipelineObjectId extends AnyRef

    

37. trait Precondition extends PipelineObject

    

    A condition that must be met before the object can run.

    A condition that must be met before the object can run. The activity cannot run until all its conditions are met.

38. case class RamdomisedObjectId(seed: String) extends PipelineObjectId with Product with Serializable

    

39. case class RedshiftCopyActivity extends PipelineActivity with Product with Serializable

    

    Redshift copy activity

40. trait RedshiftCopyOption extends AnyRef

    

41. case class RedshiftDataNode extends DataNode with Product with Serializable

    

    The abstracted RedshiftDataNode

42. trait RedshiftDatabase extends Database

    

    Redshift Database Trait, to use this please extend with an object.

43. case class RedshiftUnloadActivity extends PipelineActivity with Product with Serializable

    

    Redshift unload activity

44. trait RedshiftUnloadOption extends AnyRef

    

45. case class RegExDataFormat extends DataFormat with Product with Serializable

    

    RegEx data format

46. trait ResourceObject extends PipelineObject

    

47. trait RunnableObject extends AnyRef

    

    Run time references of runnable objects

48. trait S3DataNode extends Copyable

    

49. case class S3File(id: PipelineObjectId, filePath: String, dataFormat: Option[DataFormat], preconditions: Seq[Precondition], onSuccessAlarms: Seq[SnsAlarm], onFailAlarms: Seq[SnsAlarm]) extends S3DataNode with Product with Serializable

    

    Defines data from s3

50. case class S3Folder(id: PipelineObjectId, directoryPath: String = "", dataFormat: Option[DataFormat] = None, preconditions: Seq[Precondition] = Seq(), onSuccessAlarms: Seq[SnsAlarm] = Seq(), onFailAlarms: Seq[SnsAlarm] = Seq()) extends S3DataNode with Product with Serializable

    

    Defines data from s3 directory

51. case class S3KeyExistsPrecondition extends Precondition with Product with Serializable

    

    Checks whether a key exists in an Amazon S3 data node.

52. case class S3KeyParameter(id: String, value: String, description: Option[String] = None, encrypted: Boolean = false) extends Parameter with Product with Serializable

    

53. case class S3PrefixNotEmptyPrecondition extends Precondition with Product with Serializable

    

    A precondition to check that the Amazon S3 objects with the given prefix (represented as a URI) are present.

54. case class Schedule(id: PipelineObjectId = ScheduleObjectId, start: Option[github.nscala_time.time.Imports.DateTime] = None, period: DpPeriod = 1.day, end: Option[Either[Int, github.nscala_time.time.Imports.DateTime]] = None, scheduleType: ScheduleType = Cron) extends PipelineObject with Product with Serializable

    

    Cron liked schedule that runs at defined period.

    Cron liked schedule that runs at defined period.

    Note

    If start time given is a past time, data pipeline will perform back fill from the start.

55. case class ShellCommandActivity extends PipelineActivity with Product with Serializable

    

    Shell command activity

56. case class ShellCommandPrecondition extends Precondition with Product with Serializable

    

    A Unix/Linux shell command that can be run as a precondition.

57. case class SnsAlarm extends PipelineObject with Product with Serializable

    

58. case class SparkActivity extends EmrActivity with Product with Serializable

    

    Defines a spark activity

59. case class SparkCluster extends EmrCluster with Product with Serializable

    

    Launch a Spark cluster

60. case class SparkStep(jar: String = "", mainClass: String = "", args: Seq[String] = List())(implicit hc: HyperionContext) extends Product with Serializable

    

    A spark step that runs on Spark Cluster

61. case class SqlActivity extends PipelineActivity with Product with Serializable

    

62. case class SqlDataNode(id: PipelineObjectId, tableQuery: TableQuery, database: JdbcDatabase, preconditions: Seq[Precondition], onSuccessAlarms: Seq[SnsAlarm], onFailAlarms: Seq[SnsAlarm]) extends Copyable with Product with Serializable

    
    Note

    that the AWS Datapipeline SqlDataNode does not require a JdbcDatabase parameter, but requires specify the username, password, etc. within the object, we require a JdbcDatabase object for consistency with other database data node objects.

63. case class StringParameter(id: String, value: String, description: Option[String] = None, allowedValues: Seq[String] = Seq(), encrypted: Boolean = false) extends Parameter with Product with Serializable

    

64. case class TsvDataFormat extends DataFormat with Product with Serializable

    

    TSV data format