com.ebiznext.comet.job.index.bqload
Saves a dataset.
Saves a dataset. If the path is empty (the first time we call metrics on the schema) then we can write.
If there's already parquet files stored in it, then create a temporary directory to compute on, and flush the path to move updated metrics in it
: dataset to be saved
: Path to save the file at
To set access control on a table or view, we can use Identity and Access Management (IAM) policy After you create a table or view, you can set its policy with a set-iam-policy call For each call, we compare if the existing policy is equal to the defined one (in the Yaml file) If it's the case, we do nothing, otherwise we update the Table policy
To set access control on a table or view, we can use Identity and Access Management (IAM) policy After you create a table or view, you can set its policy with a set-iam-policy call For each call, we compare if the existing policy is equal to the defined one (in the Yaml file) If it's the case, we do nothing, otherwise we update the Table policy
Partition a dataset using dataset columns.
Partition a dataset using dataset columns. To partition the dataset using the ingestion time, use the reserved column names :
: Input dataset
: list of columns to use for partitioning.
The Spark session used to run this job
Just to force any spark job to implement its entry point within the "run" method
Just to force any spark job to implement its entry point within the "run" method
: Spark Session used for the job