com.google.cloud.hadoop.io.bigquery

Class BigQueryHelper

java.lang.Object

com.google.cloud.hadoop.io.bigquery.BigQueryHelper

public class BigQueryHelper
extends Object

Wrapper for BigQuery API.

Field Summary

Fields

Modifier and Type	Field and Description
static int	BIGQUERY_JOB_ID_MAX_LENGTH
static String	BIGQUERY_JOB_ID_PATTERN

Constructor Summary

Constructors

Constructor and Description
BigQueryHelper(com.google.api.services.bigquery.Bigquery service)

Method Summary

Modifier and Type	Method and Description
void	checkJobIdEquality(com.google.api.services.bigquery.model.Job expected, com.google.api.services.bigquery.model.Job actual) Helper to check for non-null Job.getJobReference().getJobId() and quality of the getJobId() between expected and actual, using Preconditions.checkState.
com.google.api.services.bigquery.model.JobReference	createJobReference(String projectId, String jobIdPrefix, String location) Creates a new JobReference with a unique jobId generated from jobIdPrefix plus a randomly generated UUID String.
void	exportBigQueryToGcs(String projectId, com.google.api.services.bigquery.model.TableReference tableRef, List<String> gcsPaths, boolean awaitCompletion) Exports BigQuery results into GCS, polls for completion before returning.
com.google.api.services.bigquery.Bigquery	getRawBigquery() Returns the underlying Bigquery instance used for communicating with the BigQuery API.
com.google.api.services.bigquery.model.Table	getTable(com.google.api.services.bigquery.model.TableReference tableRef) Gets the specified table resource by table ID.
void	importFederatedFromGcs(String projectId, com.google.api.services.bigquery.model.TableReference tableRef, com.google.api.services.bigquery.model.TableSchema schema, BigQueryFileFormat sourceFormat, List<String> gcsPaths) Performs a federated import on data from GCS into BigQuery via a table insert.
void	importFromGcs(String projectId, com.google.api.services.bigquery.model.TableReference tableRef, com.google.api.services.bigquery.model.TableSchema schema, com.google.api.services.bigquery.model.TimePartitioning timePartitioning, String kmsKeyName, BigQueryFileFormat sourceFormat, String createDisposition, String writeDisposition, List<String> gcsPaths, boolean awaitCompletion) Imports data from GCS into BigQuery via a load job.
com.google.api.services.bigquery.model.Job	insertJobOrFetchDuplicate(String projectId, com.google.api.services.bigquery.model.Job job) Tries to run jobs().insert(...) with the provided projectId and job, which returns a Job under normal operation, which is then returned from this method.
boolean	tableExists(com.google.api.services.bigquery.model.TableReference tableRef) Returns true if the table exists, or false if not.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

BIGQUERY_JOB_ID_PATTERN

public static final String BIGQUERY_JOB_ID_PATTERN

See Also:

Constant Field Values

BIGQUERY_JOB_ID_MAX_LENGTH

public static final int BIGQUERY_JOB_ID_MAX_LENGTH

See Also:

Constant Field Values

Constructor Detail

BigQueryHelper

public BigQueryHelper(com.google.api.services.bigquery.Bigquery service)

Method Detail

getRawBigquery

public com.google.api.services.bigquery.Bigquery getRawBigquery()

Returns the underlying Bigquery instance used for communicating with the BigQuery API.

importFederatedFromGcs

public void importFederatedFromGcs(String projectId,
                                   com.google.api.services.bigquery.model.TableReference tableRef,
                                   @Nullable
                                   com.google.api.services.bigquery.model.TableSchema schema,
                                   BigQueryFileFormat sourceFormat,
                                   List<String> gcsPaths)
                            throws IOException

Performs a federated import on data from GCS into BigQuery via a table insert.

Parameters:

projectId - the project on whose behalf to perform the load.

tableRef - the reference to the destination table.

schema - the schema of the source data to populate the destination table by.

sourceFormat - the file format of the source data.

gcsPaths - the location of the source data in GCS.

Throws:

IOException

importFromGcs

public void importFromGcs(String projectId,
                          com.google.api.services.bigquery.model.TableReference tableRef,
                          @Nullable
                          com.google.api.services.bigquery.model.TableSchema schema,
                          @Nullable
                          com.google.api.services.bigquery.model.TimePartitioning timePartitioning,
                          @Nullable
                          String kmsKeyName,
                          BigQueryFileFormat sourceFormat,
                          String createDisposition,
                          String writeDisposition,
                          List<String> gcsPaths,
                          boolean awaitCompletion)
                   throws IOException,
                          InterruptedException

Imports data from GCS into BigQuery via a load job. Optionally polls for completion before returning.

Parameters:

projectId - the project on whose behalf to perform the load.

tableRef - the reference to the destination table.

schema - the schema of the source data to populate the destination table by.

timePartitioning - time partitioning to populate the destination table.

kmsKeyName - the Cloud KMS encryption key used to protect the output table.

sourceFormat - the file format of the source data.

createDisposition - the create disposition of the output table.

writeDisposition - the write disposition of the output table.

gcsPaths - the location of the source data in GCS.

awaitCompletion - if true, block and poll until job completes, otherwise return as soon as the job has been successfully dispatched.

Throws:

IOException

InterruptedException - if interrupted while waiting for job completion.

exportBigQueryToGcs

public void exportBigQueryToGcs(String projectId,
                                com.google.api.services.bigquery.model.TableReference tableRef,
                                List<String> gcsPaths,
                                boolean awaitCompletion)
                         throws IOException,
                                InterruptedException

Exports BigQuery results into GCS, polls for completion before returning.

Parameters:

projectId - the project on whose behalf to perform the export.

tableRef - the table to export.

gcsPaths - the GCS paths to export to.

awaitCompletion - if true, block and poll until job completes, otherwise return as soon as the job has been successfully dispatched.

Throws:

IOException - on IO error.

InterruptedException - on interrupt.

tableExists

public boolean tableExists(com.google.api.services.bigquery.model.TableReference tableRef)
                    throws IOException

Returns true if the table exists, or false if not.

Throws:

IOException

getTable

public com.google.api.services.bigquery.model.Table getTable(com.google.api.services.bigquery.model.TableReference tableRef)
                                                      throws IOException

Gets the specified table resource by table ID. This method does not return the data in the table, it only returns the table resource, which describes the structure of this table.

Parameters:

tableRef - The BigQuery table reference.

Returns:

The table resource, which describes the structure of this table.

Throws:

IOException

createJobReference

public com.google.api.services.bigquery.model.JobReference createJobReference(String projectId,
                                                                              String jobIdPrefix,
                                                                              @Nullable
                                                                              String location)

Creates a new JobReference with a unique jobId generated from jobIdPrefix plus a randomly generated UUID String.

checkJobIdEquality

public void checkJobIdEquality(com.google.api.services.bigquery.model.Job expected,
                               com.google.api.services.bigquery.model.Job actual)

Helper to check for non-null Job.getJobReference().getJobId() and quality of the getJobId() between expected and actual, using Preconditions.checkState.

insertJobOrFetchDuplicate

public com.google.api.services.bigquery.model.Job insertJobOrFetchDuplicate(String projectId,
                                                                            com.google.api.services.bigquery.model.Job job)
                                                                     throws IOException

Tries to run jobs().insert(...) with the provided projectId and job, which returns a Job under normal operation, which is then returned from this method. In case of an exception being thrown, if the cause was "409 conflict", then we issue a separate "jobs().get(...)" request and return the results of that fetch instead. Other exceptions propagate out as normal.

Throws:

IOException