com.google.cloud.hadoop.io.bigquery

Class ShardedInputSplit

java.lang.Object

org.apache.hadoop.mapreduce.InputSplit

com.google.cloud.hadoop.io.bigquery.ShardedInputSplit

All Implemented Interfaces:

org.apache.hadoop.io.Writable

public class ShardedInputSplit
extends org.apache.hadoop.mapreduce.InputSplit
implements org.apache.hadoop.io.Writable

ShardedInputSplit implements Hadoop InputSplit by storing path information for a logical "shard" corresponding to a single Hadoop task. This shard will own a single subdirectory of GCS so read in dynamic ordering as a BigQuery export proceeds simultaneously.

Constructor Summary

Constructors

Constructor and Description
ShardedInputSplit() Default constructor for dynamic-classloading deserialization.
ShardedInputSplit(org.apache.hadoop.fs.Path shardDirectoryAndPattern, long estimatedNumRecords) For creation of splits in JobClient.

Method Summary

Modifier and Type	Method and Description
long	getLength() Estimated number of records to read, *not* the number of bytes.
String[]	getLocations()
org.apache.hadoop.fs.Path	getShardDirectoryAndPattern() Accessor for shardDirectoryAndPattern.
void	readFields(DataInput in)
String	toString()
void	write(DataOutput out)

Methods inherited from class org.apache.hadoop.mapreduce.InputSplit

getLocationInfo

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

ShardedInputSplit

public ShardedInputSplit()

Default constructor for dynamic-classloading deserialization.

ShardedInputSplit

public ShardedInputSplit(org.apache.hadoop.fs.Path shardDirectoryAndPattern,
                         long estimatedNumRecords)

For creation of splits in JobClient.

Method Detail

getShardDirectoryAndPattern

public org.apache.hadoop.fs.Path getShardDirectoryAndPattern()

Accessor for shardDirectoryAndPattern.

getLength

public long getLength()

Estimated number of records to read, *not* the number of bytes.

Specified by:

getLength in class org.apache.hadoop.mapreduce.InputSplit

getLocations

public String[] getLocations()
                      throws IOException

Specified by:

getLocations in class org.apache.hadoop.mapreduce.InputSplit

Throws:

IOException

toString

public String toString()

Overrides:

toString in class Object

write

public void write(DataOutput out)
           throws IOException

Specified by:

write in interface org.apache.hadoop.io.Writable

Throws:

IOException

readFields

public void readFields(DataInput in)
                throws IOException

Specified by:

readFields in interface org.apache.hadoop.io.Writable

Throws:

IOException