org.elasticsearch.hadoop.mr

Class EsInputFormat<K,V>

java.lang.Object

org.apache.hadoop.mapreduce.InputFormat<K,V>

org.elasticsearch.hadoop.mr.EsInputFormat<K,V>

All Implemented Interfaces:

org.apache.hadoop.mapred.InputFormat<K,V>

Direct Known Subclasses:

EsHiveInputFormat, EsPigInputFormat

public class EsInputFormat<K,V>
extends org.apache.hadoop.mapreduce.InputFormat<K,V>
implements org.apache.hadoop.mapred.InputFormat<K,V>

ElasticSearch InputFormat for streaming data (typically based on a query) from ElasticSearch. Returns the document ID as key and its content as value.

This class implements both the "old" (org.apache.hadoop.mapred) and the "new" (org.apache.hadoop.mapreduce) API.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	EsInputFormat.AbstractWritableEsInputRecordReader<V>
protected static class	EsInputFormat.EsInputRecordReader<K,V>
protected static class	EsInputFormat.EsInputSplit
protected static class	EsInputFormat.JsonWritableEsInputRecordReader
protected static class	EsInputFormat.WritableEsInputRecordReader

Constructor Summary

Constructors

Constructor and Description
EsInputFormat()

Method Summary

Modifier and Type	Method and Description
EsInputFormat.EsInputRecordReader<K,V>	createRecordReader(org.apache.hadoop.mapreduce.InputSplit split, org.apache.hadoop.mapreduce.TaskAttemptContext context)
EsInputFormat.EsInputRecordReader<K,V>	getRecordReader(org.apache.hadoop.mapred.InputSplit split, org.apache.hadoop.mapred.JobConf job, org.apache.hadoop.mapred.Reporter reporter)
org.apache.hadoop.mapred.InputSplit[]	getSplits(org.apache.hadoop.mapred.JobConf job, int numSplits)
java.util.List<org.apache.hadoop.mapreduce.InputSplit>	getSplits(org.apache.hadoop.mapreduce.JobContext context)
protected boolean	isOutputAsJson(org.apache.hadoop.conf.Configuration cfg)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

EsInputFormat

public EsInputFormat()

Method Detail

getSplits

public java.util.List<org.apache.hadoop.mapreduce.InputSplit> getSplits(org.apache.hadoop.mapreduce.JobContext context)
                                                                 throws java.io.IOException

Specified by:

getSplits in class org.apache.hadoop.mapreduce.InputFormat<K,V>

Throws:

java.io.IOException

createRecordReader

public EsInputFormat.EsInputRecordReader<K,V> createRecordReader(org.apache.hadoop.mapreduce.InputSplit split,
                                                                 org.apache.hadoop.mapreduce.TaskAttemptContext context)

Specified by:

createRecordReader in class org.apache.hadoop.mapreduce.InputFormat<K,V>

getSplits

public org.apache.hadoop.mapred.InputSplit[] getSplits(org.apache.hadoop.mapred.JobConf job,
                                                       int numSplits)
                                                throws java.io.IOException

Specified by:

getSplits in interface org.apache.hadoop.mapred.InputFormat<K,V>

Throws:

java.io.IOException

getRecordReader

public EsInputFormat.EsInputRecordReader<K,V> getRecordReader(org.apache.hadoop.mapred.InputSplit split,
                                                              org.apache.hadoop.mapred.JobConf job,
                                                              org.apache.hadoop.mapred.Reporter reporter)

Specified by:

getRecordReader in interface org.apache.hadoop.mapred.InputFormat<K,V>

isOutputAsJson

protected boolean isOutputAsJson(org.apache.hadoop.conf.Configuration cfg)