org.deeplearning4j.text.tokenization.tokenizerfactory

Class DefaultTokenizerFactory

java.lang.Object

org.deeplearning4j.text.tokenization.tokenizerfactory.DefaultTokenizerFactory

All Implemented Interfaces:

TokenizerFactory

public class DefaultTokenizerFactory
extends Object
implements TokenizerFactory

Default tokenizer based on string tokenizer or stream tokenizer

Author:

Adam Gibson

Constructor Summary

Constructors

Constructor and Description
DefaultTokenizerFactory()

Method Summary

Modifier and Type	Method and Description
Tokenizer	create(InputStream toTokenize) Create a tokenizer based on an input stream
Tokenizer	create(String toTokenize) The tokenizer to createComplex
TokenPreProcess	getTokenPreProcessor() Returns TokenPreProcessor set for this TokenizerFactory instance
void	setTokenPreProcessor(TokenPreProcess preProcessor) Sets a token pre processor to be used with every tokenizer

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DefaultTokenizerFactory

public DefaultTokenizerFactory()

Method Detail

create

public Tokenizer create(String toTokenize)

Description copied from interface: TokenizerFactory

The tokenizer to createComplex

Specified by:

create in interface TokenizerFactory

Parameters:

toTokenize - the string to createComplex the tokenizer with

Returns:

the new tokenizer

create

public Tokenizer create(InputStream toTokenize)

Description copied from interface: TokenizerFactory

Create a tokenizer based on an input stream

Specified by:

create in interface TokenizerFactory

Returns:

setTokenPreProcessor

public void setTokenPreProcessor(TokenPreProcess preProcessor)

Description copied from interface: TokenizerFactory

Sets a token pre processor to be used with every tokenizer

Specified by:

setTokenPreProcessor in interface TokenizerFactory

Parameters:

preProcessor - the token pre processor to use

getTokenPreProcessor

public TokenPreProcess getTokenPreProcessor()

Returns TokenPreProcessor set for this TokenizerFactory instance

Specified by:

getTokenPreProcessor in interface TokenizerFactory

Returns:

TokenPreProcessor instance, or null if no preprocessor was defined