com.google.cloud.hadoop.io.bigquery

Class BigQueryOutputFormat<K,V extends com.google.gson.JsonObject>

java.lang.Object

org.apache.hadoop.mapreduce.OutputFormat<K,V>

com.google.cloud.hadoop.io.bigquery.BigQueryOutputFormat<K,V>

Type Parameters:

K - Key type.

V - Value type must be JsonObject or a derived type..

Deprecated.

Use IndirectBigQueryOutputFormat instead.

An OutputFormat that sends the output of a Hadoop job directly to BigQuery.

BigQueryOutputFormat accepts key, value pairs, but the returned BigQueryRecordWriter writes only the value to the database as each BigQuery value already contains a BigQuery key.

This is deprecated because it creates many temporary tables, which can drain user quota.

@Deprecated
public class BigQueryOutputFormat<K,V extends com.google.gson.JsonObject>
extends org.apache.hadoop.mapreduce.OutputFormat<K,V>

Field Summary

Fields

Modifier and Type	Field and Description
static java.text.NumberFormat	NUMBER_FORMAT Deprecated.
static java.lang.String	TEMP_NAME Deprecated.

Constructor Summary

Constructors

Constructor and Description
BigQueryOutputFormat() Deprecated.

Method Summary

Modifier and Type	Method and Description
void	checkOutputSpecs(org.apache.hadoop.mapreduce.JobContext context) Deprecated. Checks for validity of the output-specification for the job.
BigQueryOutputCommitter	getOutputCommitter(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.mapreduce.TaskAttemptID taskAttemptId) Deprecated.
BigQueryOutputCommitter	getOutputCommitter(org.apache.hadoop.mapreduce.TaskAttemptContext context) Deprecated. Gets the OutputCommiter which ensures that the temporary files are cleaned up and output commits are scheduled.
org.apache.hadoop.mapreduce.RecordWriter<K,V>	getRecordWriter(org.apache.hadoop.mapreduce.TaskAttemptContext context) Deprecated. Returns a new RecordWriter for writing outputs to the BigQuery.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

NUMBER_FORMAT

public static final java.text.NumberFormat NUMBER_FORMAT

Deprecated.

TEMP_NAME

public static final java.lang.String TEMP_NAME

Deprecated.

See Also:

Constant Field Values

Constructor Detail

BigQueryOutputFormat

public BigQueryOutputFormat()

Deprecated.

Method Detail

checkOutputSpecs

public void checkOutputSpecs(org.apache.hadoop.mapreduce.JobContext context)
                      throws java.io.IOException

Deprecated.

Checks for validity of the output-specification for the job. Typically checks that it does not already exist, throwing an exception when it already exists, so that output is not overwritten. However as we only add entities to a table and are not overwriting, this method only checks if the fields, projectId, tableId, datasetId, are not null or empty and if the numRecordsInBatch is a positive int.

TODO(user): check fields is a properly formatted TableSchema.

Specified by:

checkOutputSpecs in class org.apache.hadoop.mapreduce.OutputFormat<K,V extends com.google.gson.JsonObject>

Parameters:

context - the job's context.

Throws:

java.io.IOException - on IO Error.

getOutputCommitter

public BigQueryOutputCommitter getOutputCommitter(org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                           throws java.io.IOException,
                                                  java.lang.InterruptedException

Deprecated.

Gets the OutputCommiter which ensures that the temporary files are cleaned up and output commits are scheduled.

Specified by:

getOutputCommitter in class org.apache.hadoop.mapreduce.OutputFormat<K,V extends com.google.gson.JsonObject>

Parameters:

context - the task's context.

Throws:

java.lang.InterruptedException - on Interrupt.

java.io.IOException - on IO Error.

getOutputCommitter

public BigQueryOutputCommitter getOutputCommitter(org.apache.hadoop.conf.Configuration configuration,
                                                  org.apache.hadoop.mapreduce.TaskAttemptID taskAttemptId)
                                           throws java.io.IOException,
                                                  java.lang.InterruptedException

Deprecated.

Throws:

java.io.IOException

java.lang.InterruptedException

getRecordWriter

public org.apache.hadoop.mapreduce.RecordWriter<K,V> getRecordWriter(org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                                              throws java.io.IOException

Deprecated.

Returns a new RecordWriter for writing outputs to the BigQuery.

Specified by:

getRecordWriter in class org.apache.hadoop.mapreduce.OutputFormat<K,V extends com.google.gson.JsonObject>

Parameters:

context - the task's context.

Throws:

java.io.IOException - on IOError.