DPJob

gcp4zio.dp.DPJob
See theDPJob companion object
trait DPJob

Attributes

Companion:
object
Graph
Supertypes
class Object
trait Matchable
class Any
Known subtypes
class DPJobImpl

Members list

Concise view

Value members

Abstract methods

def submitHiveJob(query: String): Task[Job]

Submits a Hive Job in Dataproc Cluster. (This API will not wait for Job Completion, to wait for Job completion use trackJobProgress)

Submits a Hive Job in Dataproc Cluster. (This API will not wait for Job Completion, to wait for Job completion use trackJobProgress)

Attributes

query

Hive SQL query to run

def submitSparkJob(args: List[String], mainClass: String, libs: List[String], conf: Map[String, String]): Task[Job]

Submits a Spark Job in Dataproc Cluster. (This API will not wait for Job Completion, to wait for Job completion use trackJobProgress)

Submits a Spark Job in Dataproc Cluster. (This API will not wait for Job Completion, to wait for Job completion use trackJobProgress)

Attributes

args

command line arguments which will be passed to spark application

conf

Key value pair of the spark properties

libs

List of jar required to run this Spark Application (including application jar)

mainClass

Main class to run

def trackJobProgress(job: Job, interval: Duration): Task[Unit]

This API will track job until Completion

This API will track job until Completion

Attributes

interval

Specifies duration each status check repetition should be spaced from the last run

job

Dataproc Job which needs to be tracked