DPJobImpl

gcp4zio.dp.DPJobImpl
case class DPJobImpl(client: JobControllerClient, cluster: String, project: String, region: String) extends DPJob

Attributes

Graph
Supertypes
trait Serializable
trait Product
trait Equals
trait DPJob
class Object
trait Matchable
class Any

Members list

Concise view

Value members

Concrete methods

def submitHiveJob(query: String): Task[Job]

Submits a Hive Job in Dataproc Cluster. (This API will not wait for Job Completion, to wait for Job completion use trackJobProgress)

Submits a Hive Job in Dataproc Cluster. (This API will not wait for Job Completion, to wait for Job completion use trackJobProgress)

Attributes

query

Hive SQL query to run

def submitSparkJob(args: List[String], mainClass: String, libs: List[String], conf: Map[String, String]): Task[Job]

Submits a Spark Job in Dataproc Cluster. (This API will not wait for Job Completion, to wait for Job completion use trackJobProgress)

Submits a Spark Job in Dataproc Cluster. (This API will not wait for Job Completion, to wait for Job completion use trackJobProgress)

Attributes

args

command line arguments which will be passed to spark application

conf

Key value pair of the spark properties

libs

List of jar required to run this Spark Application (including application jar)

mainClass

Main class to run

def trackJobProgress(job: Job, interval: Duration): Task[Unit]

This API will track job until Completion

This API will track job until Completion

Attributes

interval

Specifies duration each status check repetition should be spaced from the last run

job

Dataproc Job which needs to be tracked

Inherited methods

def productElementNames: Iterator[String]

Attributes

Inherited from:
Product
def productIterator: Iterator[Any]

Attributes

Inherited from:
Product