com.google.cloud.hadoop.io.bigquery

Class DirectBigQueryInputFormat

java.lang.Object

org.apache.hadoop.mapreduce.InputFormat<org.apache.hadoop.io.NullWritable,org.apache.avro.generic.GenericRecord>

com.google.cloud.hadoop.io.bigquery.DirectBigQueryInputFormat

@InterfaceStability.Evolving
public class DirectBigQueryInputFormat
extends org.apache.hadoop.mapreduce.InputFormat<org.apache.hadoop.io.NullWritable,org.apache.avro.generic.GenericRecord>

InputFormat that directly reads data from BigQuery using the BigQuery Storage API. See https://cloud.google.com/bigquery/docs/reference/storage/.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DirectBigQueryInputFormat.DirectBigQueryInputSplit InputSplit containing session metadata.

Constructor Summary

Constructors

Constructor and Description
DirectBigQueryInputFormat()

Method Summary

Modifier and Type	Method and Description
org.apache.hadoop.mapreduce.RecordReader<org.apache.hadoop.io.NullWritable,org.apache.avro.generic.GenericRecord>	createRecordReader(org.apache.hadoop.mapreduce.InputSplit inputSplit, org.apache.hadoop.mapreduce.TaskAttemptContext context)
protected BigQueryHelper	getBigQueryHelper(org.apache.hadoop.conf.Configuration config) Helper method to override for testing.
protected com.google.cloud.bigquery.storage.v1beta1.BigQueryStorageClient	getClient(org.apache.hadoop.conf.Configuration config) Helper method to override for testing.
List<org.apache.hadoop.mapreduce.InputSplit>	getSplits(org.apache.hadoop.mapreduce.JobContext context)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DirectBigQueryInputFormat

public DirectBigQueryInputFormat()

Method Detail

getSplits

public List<org.apache.hadoop.mapreduce.InputSplit> getSplits(org.apache.hadoop.mapreduce.JobContext context)
                                                       throws IOException

Specified by:

getSplits in class org.apache.hadoop.mapreduce.InputFormat<org.apache.hadoop.io.NullWritable,org.apache.avro.generic.GenericRecord>

Throws:

IOException

createRecordReader

public org.apache.hadoop.mapreduce.RecordReader<org.apache.hadoop.io.NullWritable,org.apache.avro.generic.GenericRecord> createRecordReader(org.apache.hadoop.mapreduce.InputSplit inputSplit,
                                                                                                                                            org.apache.hadoop.mapreduce.TaskAttemptContext context)

Specified by:

createRecordReader in class org.apache.hadoop.mapreduce.InputFormat<org.apache.hadoop.io.NullWritable,org.apache.avro.generic.GenericRecord>

getClient

protected com.google.cloud.bigquery.storage.v1beta1.BigQueryStorageClient getClient(org.apache.hadoop.conf.Configuration config)
                                                                             throws IOException

Helper method to override for testing.

Returns:

Bigquery.

Throws:

IOException - on IO Error.

getBigQueryHelper

protected BigQueryHelper getBigQueryHelper(org.apache.hadoop.conf.Configuration config)
                                    throws GeneralSecurityException,
                                           IOException

Helper method to override for testing.

Throws:

GeneralSecurityException

IOException