com.google.cloud.hadoop.io.bigquery

Class AbstractBigQueryInputFormat<K,V>

java.lang.Object

org.apache.hadoop.mapreduce.InputFormat<K,V>

com.google.cloud.hadoop.io.bigquery.AbstractBigQueryInputFormat<K,V>

Type Parameters:

K - Key type

V - Value type

All Implemented Interfaces:

DelegateRecordReaderFactory<K,V>

Direct Known Subclasses:

AvroBigQueryInputFormat, GsonBigQueryInputFormat, JsonTextBigQueryInputFormat

public abstract class AbstractBigQueryInputFormat<K,V>
extends org.apache.hadoop.mapreduce.InputFormat<K,V>
implements DelegateRecordReaderFactory<K,V>

Abstract base class for BigQuery input formats. This class is expected to take care of performing BigQuery exports to temporary tables, BigQuery exports to GCS and cleaning up any files or tables that either of those processes create.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	EXTERNAL_TABLE_TYPE The keyword for the type of BigQueryTable store externally.
static java.lang.String	INPUT_FORMAT_CLASS_KEY Configuration key for InputFormat class name.

Constructor Summary

Constructors

Constructor and Description
AbstractBigQueryInputFormat()

Method Summary

Modifier and Type	Method and Description
static void	cleanupJob(BigQueryHelper bigQueryHelper, org.apache.hadoop.conf.Configuration config) Similar to cleanupJob(JobContext), but allows specifying the Bigquery instance to use.
static void	cleanupJob(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.mapreduce.JobID jobId) Cleans up relevant temporary resources associated with a job which used the GsonBigQueryInputFormat; this should be called explicitly after the completion of the entire job.
static void	cleanupJob(org.apache.hadoop.mapreduce.JobContext context) Deprecated. Use cleanupJob(Configuration, JobID)
org.apache.hadoop.mapreduce.RecordReader<K,V>	createRecordReader(org.apache.hadoop.mapreduce.InputSplit inputSplit, org.apache.hadoop.conf.Configuration configuration)
org.apache.hadoop.mapreduce.RecordReader<K,V>	createRecordReader(org.apache.hadoop.mapreduce.InputSplit inputSplit, org.apache.hadoop.mapreduce.TaskAttemptContext taskAttemptContext)
protected com.google.api.services.bigquery.Bigquery	getBigQuery(org.apache.hadoop.conf.Configuration config) Helper method to override for testing.
protected BigQueryHelper	getBigQueryHelper(org.apache.hadoop.conf.Configuration config) Helper method to override for testing.
abstract ExportFileFormat	getExportFileFormat() Get the ExportFileFormat that this input format supports.
protected static ExportFileFormat	getExportFileFormat(java.lang.Class<? extends AbstractBigQueryInputFormat<?,?>> clazz)
protected static ExportFileFormat	getExportFileFormat(org.apache.hadoop.conf.Configuration configuration)
java.util.List<org.apache.hadoop.mapreduce.InputSplit>	getSplits(org.apache.hadoop.mapreduce.JobContext context)
protected static boolean	isShardedExportEnabled(org.apache.hadoop.conf.Configuration configuration)
static void	setEnableShardedExport(org.apache.hadoop.conf.Configuration configuration, boolean enabled) Enable or disable BigQuery sharded output.
static void	setInputTable(org.apache.hadoop.conf.Configuration configuration, java.lang.String projectId, java.lang.String datasetId, java.lang.String tableId) Configure the BigQuery input table for a job
static void	setInputTable(org.apache.hadoop.conf.Configuration configuration, com.google.api.services.bigquery.model.TableReference tableReference) Configure the BigQuery input table for a job
static void	setTemporaryCloudStorageDirectory(org.apache.hadoop.conf.Configuration configuration, java.lang.String path) Configure a directory to which we will export BigQuery data

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.google.cloud.hadoop.io.bigquery.DelegateRecordReaderFactory

createDelegateRecordReader

Field Detail

INPUT_FORMAT_CLASS_KEY

public static final java.lang.String INPUT_FORMAT_CLASS_KEY

Configuration key for InputFormat class name.

See Also:

Constant Field Values

EXTERNAL_TABLE_TYPE

public static final java.lang.String EXTERNAL_TABLE_TYPE

The keyword for the type of BigQueryTable store externally.

See Also:

Constant Field Values

Constructor Detail

AbstractBigQueryInputFormat

public AbstractBigQueryInputFormat()

Method Detail

setInputTable

public static void setInputTable(org.apache.hadoop.conf.Configuration configuration,
                                 java.lang.String projectId,
                                 java.lang.String datasetId,
                                 java.lang.String tableId)
                          throws java.io.IOException

Configure the BigQuery input table for a job

Throws:

java.io.IOException

setInputTable

public static void setInputTable(org.apache.hadoop.conf.Configuration configuration,
                                 com.google.api.services.bigquery.model.TableReference tableReference)
                          throws java.io.IOException

Configure the BigQuery input table for a job

Throws:

java.io.IOException

setTemporaryCloudStorageDirectory

public static void setTemporaryCloudStorageDirectory(org.apache.hadoop.conf.Configuration configuration,
                                                     java.lang.String path)

Configure a directory to which we will export BigQuery data

setEnableShardedExport

public static void setEnableShardedExport(org.apache.hadoop.conf.Configuration configuration,
                                          boolean enabled)

Enable or disable BigQuery sharded output.

isShardedExportEnabled

protected static boolean isShardedExportEnabled(org.apache.hadoop.conf.Configuration configuration)

getExportFileFormat

public abstract ExportFileFormat getExportFileFormat()

Get the ExportFileFormat that this input format supports.

getSplits

public java.util.List<org.apache.hadoop.mapreduce.InputSplit> getSplits(org.apache.hadoop.mapreduce.JobContext context)
                                                                 throws java.io.IOException,
                                                                        java.lang.InterruptedException

Specified by:

getSplits in class org.apache.hadoop.mapreduce.InputFormat<K,V>

Throws:

java.io.IOException

java.lang.InterruptedException

createRecordReader

public org.apache.hadoop.mapreduce.RecordReader<K,V> createRecordReader(org.apache.hadoop.mapreduce.InputSplit inputSplit,
                                                                        org.apache.hadoop.mapreduce.TaskAttemptContext taskAttemptContext)
                                                                 throws java.io.IOException,
                                                                        java.lang.InterruptedException

Specified by:

createRecordReader in class org.apache.hadoop.mapreduce.InputFormat<K,V>

Throws:

java.io.IOException

java.lang.InterruptedException

createRecordReader

public org.apache.hadoop.mapreduce.RecordReader<K,V> createRecordReader(org.apache.hadoop.mapreduce.InputSplit inputSplit,
                                                                        org.apache.hadoop.conf.Configuration configuration)
                                                                 throws java.io.IOException,
                                                                        java.lang.InterruptedException

Throws:

java.io.IOException

java.lang.InterruptedException

cleanupJob

@Deprecated
public static void cleanupJob(org.apache.hadoop.mapreduce.JobContext context)
                                   throws java.io.IOException

Deprecated. Use cleanupJob(Configuration, JobID)

Cleans up relevant temporary resources associated with a job which used the GsonBigQueryInputFormat; this should be called explicitly after the completion of the entire job. Possibly cleans up intermediate export tables if configured to use one due to specifying a BigQuery "query" for the input. Cleans up the GCS directoriy where BigQuery exported its files for reading.

Parameters:

context - The JobContext which contains the full configuration plus JobID which matches the JobContext seen in the corresponding BigQueryInptuFormat.getSplits() setup.

Throws:

java.io.IOException

cleanupJob

public static void cleanupJob(org.apache.hadoop.conf.Configuration configuration,
                              org.apache.hadoop.mapreduce.JobID jobId)
                       throws java.io.IOException

Cleans up relevant temporary resources associated with a job which used the GsonBigQueryInputFormat; this should be called explicitly after the completion of the entire job. Possibly cleans up intermediate export tables if configured to use one due to specifying a BigQuery "query" for the input. Cleans up the GCS directoriy where BigQuery exported its files for reading.

Throws:

java.io.IOException

cleanupJob

public static void cleanupJob(BigQueryHelper bigQueryHelper,
                              org.apache.hadoop.conf.Configuration config)
                       throws java.io.IOException

Similar to cleanupJob(JobContext), but allows specifying the Bigquery instance to use.

Parameters:

bigQueryHelper - The Bigquery API-client helper instance to use.

config - The job Configuration object which contains settings such as whether sharded export was enabled, which GCS directory the export was performed in, etc.

Throws:

java.io.IOException

getExportFileFormat

protected static ExportFileFormat getExportFileFormat(org.apache.hadoop.conf.Configuration configuration)

getExportFileFormat

protected static ExportFileFormat getExportFileFormat(java.lang.Class<? extends AbstractBigQueryInputFormat<?,?>> clazz)

getBigQuery

protected com.google.api.services.bigquery.Bigquery getBigQuery(org.apache.hadoop.conf.Configuration config)
                                                         throws java.security.GeneralSecurityException,
                                                                java.io.IOException

Helper method to override for testing.

Returns:

Bigquery.

Throws:

java.io.IOException - on IO Error.

java.security.GeneralSecurityException - on security exception.

getBigQueryHelper

protected BigQueryHelper getBigQueryHelper(org.apache.hadoop.conf.Configuration config)
                                    throws java.security.GeneralSecurityException,
                                           java.io.IOException

Helper method to override for testing.

Throws:

java.security.GeneralSecurityException

java.io.IOException