org.apache.flink.api.java.io

Class CommonCsvInputFormat<OUT>

java.lang.Object

org.apache.flink.api.common.io.RichInputFormat<OT,org.apache.flink.core.fs.FileInputSplit>

org.apache.flink.api.common.io.FileInputFormat<OT>

org.apache.flink.api.common.io.DelimitedInputFormat<OT>

org.apache.flink.api.common.io.GenericCsvInputFormat<OUT>

org.apache.flink.api.java.io.CommonCsvInputFormat<OUT>

All Implemented Interfaces:

Serializable, org.apache.flink.api.common.io.InputFormat<OUT,org.apache.flink.core.fs.FileInputSplit>, org.apache.flink.core.io.InputSplitSource<org.apache.flink.core.fs.FileInputSplit>

Direct Known Subclasses:

CsvInputFormat

public abstract class CommonCsvInputFormat<OUT>
extends org.apache.flink.api.common.io.GenericCsvInputFormat<OUT>

See Also:

Serialized Form

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.flink.api.common.io.DelimitedInputFormat

org.apache.flink.api.common.io.DelimitedInputFormat.AbstractConfigBuilder<T>, org.apache.flink.api.common.io.DelimitedInputFormat.ConfigBuilder

Nested classes/interfaces inherited from class org.apache.flink.api.common.io.FileInputFormat

org.apache.flink.api.common.io.FileInputFormat.FileBaseStatistics, org.apache.flink.api.common.io.FileInputFormat.InputSplitOpenThread

Field Summary

Fields

Modifier and Type	Field and Description
static String	DEFAULT_FIELD_DELIMITER
static String	DEFAULT_LINE_DELIMITER
protected Object[]	parsedValues

Fields inherited from class org.apache.flink.api.common.io.GenericCsvInputFormat

commentCount, commentPrefix, fieldIncluded, invalidLineCount, lineDelimiterIsLinebreak

Fields inherited from class org.apache.flink.api.common.io.DelimitedInputFormat

RECORD_DELIMITER

Fields inherited from class org.apache.flink.api.common.io.FileInputFormat

currentSplit, ENUMERATE_NESTED_FILES_FLAG, enumerateNestedFiles, filePath, INFLATER_INPUT_STREAM_FACTORIES, minSplitSize, numSplits, openTimeout, READ_WHOLE_SPLIT_FLAG, splitLength, splitStart, stream, unsplittable

Constructor Summary

Constructors

Constructor and Description
CommonCsvInputFormat(org.apache.flink.core.fs.Path filePath, org.apache.flink.api.common.typeutils.CompositeType<OUT> typeInformation)
CommonCsvInputFormat(org.apache.flink.core.fs.Path filePath, String lineDelimiter, String fieldDelimiter, org.apache.flink.api.common.typeutils.CompositeType<OUT> compositeTypeInfo)

Method Summary

Modifier and Type	Method and Description
protected abstract OUT	createTuple(OUT reuse)
Class<?>[]	getFieldTypes()
OUT	nextRecord(OUT record)
void	open(org.apache.flink.core.fs.FileInputSplit split)
OUT	readRecord(OUT reuse, byte[] bytes, int offset, int numBytes)
void	setFields(boolean[] sourceFieldMask, Class<?>[] fieldTypes)
void	setFields(int[] sourceFieldIndices, Class<?>[] fieldTypes)
void	setFieldTypes(Class<?>... fieldTypes)
void	setOrderOfPOJOFields(String[] fieldNames)

Methods inherited from class org.apache.flink.api.common.io.GenericCsvInputFormat

checkAndCoSort, checkForMonotonousOrder, close, enableQuotedStringParsing, getCommentPrefix, getFieldDelimiter, getFieldParsers, getGenericFieldTypes, getNumberOfFieldsTotal, getNumberOfNonNullFields, isLenient, isSkippingFirstLineAsHeader, parseRecord, setCommentPrefix, setCommentPrefix, setCommentPrefix, setCommentPrefix, setCommentPrefix, setFieldDelimiter, setFieldDelimiter, setFieldDelimiter, setFieldsGeneric, setFieldsGeneric, setFieldTypesGeneric, setLenient, setSkipFirstLineAsHeader, skipFields

Methods inherited from class org.apache.flink.api.common.io.DelimitedInputFormat

configure, configureDelimitedFormat, getBufferSize, getDelimiter, getLineLengthLimit, getNumLineSamples, getStatistics, loadGloablConfigParams, reachedEnd, readLine, setBufferSize, setDelimiter, setDelimiter, setDelimiter, setLineLengthLimit, setNumLineSamples

Methods inherited from class org.apache.flink.api.common.io.FileInputFormat

acceptFile, configureFileFormat, createInputSplits, decorateInputStream, extractFileExtension, getFilePath, getFileStats, getInflaterInputStreamFactory, getInputSplitAssigner, getMinSplitSize, getNumSplits, getOpenTimeout, getSplitLength, getSplitStart, registerInflaterInputStreamFactory, setFilePath, setFilePath, setMinSplitSize, setNumSplits, setOpenTimeout, testForUnsplittable, toString

Methods inherited from class org.apache.flink.api.common.io.RichInputFormat

getRuntimeContext, setRuntimeContext

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_LINE_DELIMITER

public static final String DEFAULT_LINE_DELIMITER

See Also:

Constant Field Values

DEFAULT_FIELD_DELIMITER

public static final String DEFAULT_FIELD_DELIMITER

See Also:

Constant Field Values

parsedValues

protected transient Object[] parsedValues

Constructor Detail

CommonCsvInputFormat

public CommonCsvInputFormat(org.apache.flink.core.fs.Path filePath,
                            org.apache.flink.api.common.typeutils.CompositeType<OUT> typeInformation)

CommonCsvInputFormat

public CommonCsvInputFormat(org.apache.flink.core.fs.Path filePath,
                            String lineDelimiter,
                            String fieldDelimiter,
                            org.apache.flink.api.common.typeutils.CompositeType<OUT> compositeTypeInfo)

Method Detail

setOrderOfPOJOFields

public void setOrderOfPOJOFields(String[] fieldNames)

setFieldTypes

public void setFieldTypes(Class<?>... fieldTypes)

setFields

public void setFields(int[] sourceFieldIndices,
                      Class<?>[] fieldTypes)

setFields

public void setFields(boolean[] sourceFieldMask,
                      Class<?>[] fieldTypes)

getFieldTypes

public Class<?>[] getFieldTypes()

open

public void open(org.apache.flink.core.fs.FileInputSplit split)
          throws IOException

Specified by:

open in interface org.apache.flink.api.common.io.InputFormat<OUT,org.apache.flink.core.fs.FileInputSplit>

Overrides:

open in class org.apache.flink.api.common.io.GenericCsvInputFormat<OUT>

Throws:

IOException

nextRecord

public OUT nextRecord(OUT record)
               throws IOException

Specified by:

nextRecord in interface org.apache.flink.api.common.io.InputFormat<OUT,org.apache.flink.core.fs.FileInputSplit>

Overrides:

nextRecord in class org.apache.flink.api.common.io.DelimitedInputFormat<OUT>

Throws:

IOException

readRecord

public OUT readRecord(OUT reuse,
                      byte[] bytes,
                      int offset,
                      int numBytes)
               throws IOException

Specified by:

readRecord in class org.apache.flink.api.common.io.DelimitedInputFormat<OUT>

Throws:

IOException

createTuple

protected abstract OUT createTuple(OUT reuse)