org.apache.flink.addons.hbase

Class AbstractTableInputFormat<T>

java.lang.Object

org.apache.flink.api.common.io.RichInputFormat<T,TableInputSplit>

org.apache.flink.addons.hbase.AbstractTableInputFormat<T>

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.io.InputFormat<T,TableInputSplit>, org.apache.flink.core.io.InputSplitSource<TableInputSplit>

Direct Known Subclasses:

HBaseRowInputFormat, TableInputFormat

public abstract class AbstractTableInputFormat<T>
extends org.apache.flink.api.common.io.RichInputFormat<T,TableInputSplit>

Abstract InputFormat to read data from HBase tables.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected byte[]	currentRow
protected boolean	endReached
protected static org.slf4j.Logger	LOG
protected org.apache.hadoop.hbase.client.ResultScanner	resultScanner HBase iterator wrapper.
protected org.apache.hadoop.hbase.client.Scan	scan
protected long	scannedRows
protected org.apache.hadoop.hbase.client.HTable	table

Constructor Summary

Constructors

Constructor and Description
AbstractTableInputFormat()

Method Summary

Modifier and Type	Method and Description
void	close()
void	closeInputFormat()
abstract void	configure(org.apache.flink.configuration.Configuration parameters) Creates a Scan object and opens the HTable connection.
TableInputSplit[]	createInputSplits(int minNumSplits)
org.apache.flink.core.io.InputSplitAssigner	getInputSplitAssigner(TableInputSplit[] inputSplits)
protected abstract org.apache.hadoop.hbase.client.Scan	getScanner() Returns an instance of Scan that retrieves the required subset of records from the HBase table.
org.apache.flink.api.common.io.statistics.BaseStatistics	getStatistics(org.apache.flink.api.common.io.statistics.BaseStatistics cachedStatistics)
protected abstract String	getTableName() What table is to be read.
protected boolean	includeRegionInScan(byte[] startKey, byte[] endKey) Test if the given region is to be included in the scan while splitting the regions of a table.
protected abstract T	mapResultToOutType(org.apache.hadoop.hbase.client.Result r) HBase returns an instance of Result.
T	nextRecord(T reuse)
void	open(TableInputSplit split)
boolean	reachedEnd()

Methods inherited from class org.apache.flink.api.common.io.RichInputFormat

getRuntimeContext, openInputFormat, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

protected static final org.slf4j.Logger LOG

endReached

protected boolean endReached

table

protected transient org.apache.hadoop.hbase.client.HTable table

scan

protected transient org.apache.hadoop.hbase.client.Scan scan

resultScanner

protected org.apache.hadoop.hbase.client.ResultScanner resultScanner

HBase iterator wrapper.

currentRow

protected byte[] currentRow

scannedRows

protected long scannedRows

Constructor Detail

AbstractTableInputFormat

public AbstractTableInputFormat()

Method Detail

getScanner

protected abstract org.apache.hadoop.hbase.client.Scan getScanner()

Returns an instance of Scan that retrieves the required subset of records from the HBase table.

Returns:

The appropriate instance of Scan for this use case.

getTableName

protected abstract String getTableName()

What table is to be read.

Per instance of a TableInputFormat derivative only a single table name is possible.

Returns:

The name of the table

mapResultToOutType

protected abstract T mapResultToOutType(org.apache.hadoop.hbase.client.Result r)

HBase returns an instance of Result.

This method maps the returned Result instance into the output type T.

Parameters:

r - The Result instance from HBase that needs to be converted

Returns:

The appropriate instance of T that contains the data of Result.

configure

public abstract void configure(org.apache.flink.configuration.Configuration parameters)

Creates a Scan object and opens the HTable connection.

These are opened here because they are needed in the createInputSplits which is called before the openInputFormat method.

The connection is opened in this method and closed in closeInputFormat().

Parameters:

parameters - The configuration that is to be used

See Also:

Configuration

open

public void open(TableInputSplit split)
          throws IOException

Throws:

IOException

nextRecord

public T nextRecord(T reuse)
             throws IOException

Throws:

IOException

reachedEnd

public boolean reachedEnd()
                   throws IOException

Throws:

IOException

close

public void close()
           throws IOException

Throws:

IOException

closeInputFormat

public void closeInputFormat()
                      throws IOException

Overrides:

closeInputFormat in class org.apache.flink.api.common.io.RichInputFormat<T,TableInputSplit>

Throws:

IOException

createInputSplits

public TableInputSplit[] createInputSplits(int minNumSplits)
                                    throws IOException

Throws:

IOException

includeRegionInScan

protected boolean includeRegionInScan(byte[] startKey,
                                      byte[] endKey)

Test if the given region is to be included in the scan while splitting the regions of a table.

Parameters:

startKey - Start key of the region

endKey - End key of the region

Returns:

true, if this region needs to be included as part of the input (default).

getInputSplitAssigner

public org.apache.flink.core.io.InputSplitAssigner getInputSplitAssigner(TableInputSplit[] inputSplits)

getStatistics

public org.apache.flink.api.common.io.statistics.BaseStatistics getStatistics(org.apache.flink.api.common.io.statistics.BaseStatistics cachedStatistics)