io.cdap.cdap.etl.api.engine.sql

Interface SQLEngine<KEY_IN,VALUE_IN,KEY_OUT,VALUE_OUT>

Type Parameters:

KEY_OUT - The type for the Output Key when mapping a StructuredRecord

VALUE_OUT - The type for the Output Value when mapping a StructuredRecord

KEY_IN - The type for the Input Key when building a StructuredRecord

VALUE_IN - The type for the Input Value when building a StructuredRecord

All Superinterfaces:

PipelineConfigurable, SubmitterLifecycle<RuntimeContext>

All Known Implementing Classes:

BatchSQLEngine

@Beta
public interface SQLEngine<KEY_IN,VALUE_IN,KEY_OUT,VALUE_OUT>
extends PipelineConfigurable, SubmitterLifecycle<RuntimeContext>

A SQL Engine can be used to pushdown certain dataset operations.

SQL Engines are implemented as plugins.

Internally, the SQL Engine needs to handle retries and only surface SQLEngineException errors when there is a problem that cannot be recovered from and pipeline must be stopped.

Note that this operation may be refactored at a later release, and thus this method signature is not guaranteed to remain stable.

Method Summary

Modifier and Type	Method and Description
boolean	canJoin(SQLJoinDefinition joinDefinition) Check if the supplied Join Definition can be executed in this engine.
void	cleanup(String datasetName) Deletes all temporary datasets and cleans up all temporary data from the SQL engine.
boolean	exists(String datasetName) Check if this dataset exists in the SQL Engine.
SQLPullDataset<io.cdap.cdap.api.data.format.StructuredRecord,KEY_IN,VALUE_IN>	getPullProvider(SQLPullRequest pullRequest) Creates an InputFormatProvider that can be used to pull records from the specified dataset.
SQLPushDataset<io.cdap.cdap.api.data.format.StructuredRecord,KEY_OUT,VALUE_OUT>	getPushProvider(SQLPushRequest pushRequest) Creates an Output Format Provided that can be used to push records into a SQL Engine.
SQLDataset	join(SQLJoinRequest joinRequest) Executes the join operation defined by the supplied join request.

Methods inherited from interface io.cdap.cdap.etl.api.PipelineConfigurable

configurePipeline

Methods inherited from interface io.cdap.cdap.etl.api.SubmitterLifecycle

onRunFinish, prepareRun

Method Detail

getPushProvider

SQLPushDataset<io.cdap.cdap.api.data.format.StructuredRecord,KEY_OUT,VALUE_OUT> getPushProvider(SQLPushRequest pushRequest)
                                                                                         throws SQLEngineException

Creates an Output Format Provided that can be used to push records into a SQL Engine.

After created, this dataset will be considered "locked" until the output has been committed.

Parameters:

pushRequest - the request containing information about the dataset name and schema.t

Returns:

SQLPushDataset instance that can be used to write records to the SQL Engine.

Throws:

SQLEngineException

getPullProvider

SQLPullDataset<io.cdap.cdap.api.data.format.StructuredRecord,KEY_IN,VALUE_IN> getPullProvider(SQLPullRequest pullRequest)
                                                                                       throws SQLEngineException

Creates an InputFormatProvider that can be used to pull records from the specified dataset.

Parameters:

pullRequest - the request containing information about the dataset name and schema.

Returns:

SQLPullDataset instance that can be used to read records from the SQL engine.

Throws:

SQLEngineException

exists

boolean exists(String datasetName)
        throws SQLEngineException

Check if this dataset exists in the SQL Engine.

This is a blocking call. if the process to write records into a dataset is ongoing, this method will block until the process completes. This ensures an accurate result for this operation.

Parameters:

datasetName - the dataset name.

Returns:

boolean specifying if this dataset exists in the remote engine.

Throws:

SQLEngineException

canJoin

boolean canJoin(SQLJoinDefinition joinDefinition)

Check if the supplied Join Definition can be executed in this engine.

Parameters:

joinDefinition - the join definition to validate

Returns:

boolean specifying if this join operation can be executed in the SQl Engine.

join

SQLDataset join(SQLJoinRequest joinRequest)
         throws SQLEngineException

Executes the join operation defined by the supplied join request.

All datasets involved in this joinRequest must be pushed to the SQL engine by calling the getPushProvider(SQLPushRequest) method, or as a result of another operation.

The returned SQLDataset represents the resulting record form this operation

Parameters:

joinRequest - the join request to execute.

Returns:

the SQLDataset instance representing the output of this operation.

Throws:

SQLEngineException

cleanup

void cleanup(String datasetName)
      throws SQLEngineException

Deletes all temporary datasets and cleans up all temporary data from the SQL engine.

Parameters:

datasetName - boolean specifying if all running tasks should be stopped at this time (if any are running).

Throws:

SQLEngineException