Package org.deeplearning4j.text.tokenization.tokenizerfactory

Class NGramTokenizerFactory

java.lang.Object

org.deeplearning4j.text.tokenization.tokenizerfactory.NGramTokenizerFactory

All Implemented Interfaces:

TokenizerFactory

public class NGramTokenizerFactory
extends Object
implements TokenizerFactory

Author:

sonali

Constructor Summary

Constructors

Constructor	Description
NGramTokenizerFactory(TokenizerFactory tokenizerFactory, Integer minN, Integer maxN)

Method Summary

Modifier and Type	Method	Description
Tokenizer	create(InputStream toTokenize)	Create a tokenizer based on an input stream
Tokenizer	create(String toTokenize)	The tokenizer to createComplex
TokenPreProcess	getTokenPreProcessor()	Returns TokenPreProcessor set for this TokenizerFactory instance
void	setTokenPreProcessor(TokenPreProcess preProcessor)	Sets a token pre processor to be used with every tokenizer

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

NGramTokenizerFactory

public NGramTokenizerFactory(TokenizerFactory tokenizerFactory,
                             Integer minN,
                             Integer maxN)

Method Detail

create

public Tokenizer create(String toTokenize)

Description copied from interface: TokenizerFactory

The tokenizer to createComplex

Specified by:

create in interface TokenizerFactory

Parameters:

toTokenize - the string to createComplex the tokenizer with

Returns:

the new tokenizer

create

public Tokenizer create(InputStream toTokenize)

Description copied from interface: TokenizerFactory

Create a tokenizer based on an input stream

Specified by:

create in interface TokenizerFactory

Returns:

setTokenPreProcessor

public void setTokenPreProcessor(TokenPreProcess preProcessor)

Description copied from interface: TokenizerFactory

Sets a token pre processor to be used with every tokenizer

Specified by:

setTokenPreProcessor in interface TokenizerFactory

Parameters:

preProcessor - the token pre processor to use

getTokenPreProcessor

public TokenPreProcess getTokenPreProcessor()

Returns TokenPreProcessor set for this TokenizerFactory instance

Specified by:

getTokenPreProcessor in interface TokenizerFactory

Returns:

TokenPreProcessor instance, or null if no preprocessor was defined