io.cdap.cdap.etl.api.batch

Class BatchJoiner<JOIN_KEY,INPUT_RECORD,OUT>

java.lang.Object

io.cdap.cdap.etl.api.batch.MultiInputBatchConfigurable<BatchJoinerContext>

io.cdap.cdap.etl.api.batch.BatchJoiner<JOIN_KEY,INPUT_RECORD,OUT>

Type Parameters:

JOIN_KEY - type of join key. Must be a supported type

INPUT_RECORD - type of input record. Must be a supported type

OUT - type of output object

All Implemented Interfaces:

Destroyable, Joiner<JOIN_KEY,INPUT_RECORD,OUT>, MultiInputPipelineConfigurable, StageLifecycle<BatchJoinerRuntimeContext>, SubmitterLifecycle<BatchJoinerContext>

@Beta
public abstract class BatchJoiner<JOIN_KEY,INPUT_RECORD,OUT>
extends MultiInputBatchConfigurable<BatchJoinerContext>
implements Joiner<JOIN_KEY,INPUT_RECORD,OUT>, MultiInputPipelineConfigurable, StageLifecycle<BatchJoinerRuntimeContext>

A Joiner used for batch programs. As it is used in batch programs, a BatchJoiner must be parameterized with supported join key and input record classes. Join keys and input records can be a byte[], Boolean, Integer, Long, Float, Double, String, or StructuredRecord. If the join key is not one of those types and is being used in mapreduce, it must implement Hadoop's org.apache.hadoop.io.WritableComparable interface. If the input record is not one of those types and is being used in mapreduce, it must implement Hadoop's org.apache.hadoop.io.Writable interface. If the joiner is being used in spark, both the join key and input record must implement the Serializable interface.

Field Summary

Fields

Modifier and Type	Field and Description
static String	PLUGIN_TYPE

Constructor Summary

Constructors

Constructor and Description
BatchJoiner()

Method Summary

Modifier and Type	Method and Description
void	configurePipeline(MultiInputPipelineConfigurer multiInputPipelineConfigurer) Configure the pipeline.
void	destroy() Destroy the Batch Joiner.
void	initialize(BatchJoinerRuntimeContext context) Initialize the Batch Joiner.
void	prepareRun(BatchJoinerContext context) Prepare a pipeline run.

Methods inherited from class io.cdap.cdap.etl.api.batch.MultiInputBatchConfigurable

onRunFinish

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface io.cdap.cdap.etl.api.Joiner

getJoinConfig, getJoinKeys, joinOn, merge

Field Detail

PLUGIN_TYPE

public static final String PLUGIN_TYPE

See Also:

Constant Field Values

Constructor Detail

BatchJoiner

public BatchJoiner()

Method Detail

configurePipeline

public void configurePipeline(MultiInputPipelineConfigurer multiInputPipelineConfigurer)

Configure the pipeline. This is run once when the pipeline is being published. This is where you perform any static logic, like creating required datasets, performing schema validation, setting output schema, and things of that nature.

Specified by:

configurePipeline in interface MultiInputPipelineConfigurable

Overrides:

configurePipeline in class MultiInputBatchConfigurable<BatchJoinerContext>

Parameters:

multiInputPipelineConfigurer - the configurer used to add required datasets and streams

prepareRun

public void prepareRun(BatchJoinerContext context)
                throws Exception

Prepare a pipeline run. This is run every time before a pipeline runs in order to help set up the run. This is where you would set things like the number of partitions to use when joining, and setting the join key class if they are not known at compile time.

Specified by:

prepareRun in interface SubmitterLifecycle<BatchJoinerContext>

Specified by:

prepareRun in class MultiInputBatchConfigurable<BatchJoinerContext>

Parameters:

context - batch execution context

Throws:

Exception

initialize

public void initialize(BatchJoinerRuntimeContext context)
                throws Exception

Initialize the Batch Joiner. Executed inside the Batch Run. This method is guaranteed to be invoked before any calls to Joiner.joinOn(String, Object) and Joiner.merge(Object, Iterable) are made.

Specified by:

initialize in interface StageLifecycle<BatchJoinerRuntimeContext>

Parameters:

context - runtime context for joiner which exposes input schemas and output schema for joiner

Throws:

Exception - if there is any error during initialization

destroy

public void destroy()

Destroy the Batch Joiner. Executed at the end of the Batch Run.

Specified by:

destroy in interface Destroyable