org.deeplearning4j.text.tokenization.tokenizer

Interface Tokenizer

All Known Implementing Classes:

DefaultStreamTokenizer, DefaultTokenizer, NGramTokenizer

public interface Tokenizer

A representation of a tokenizer. Different applications may require different kind of tokenization (say rules based vs more formal NLP approaches)

Author:

Adam Gibson

Method Summary

Modifier and Type	Method and Description
int	countTokens() The number of tokens in the tokenizer
List<String>	getTokens() Returns a list of all the tokens
boolean	hasMoreTokens() An iterator for tracking whether more tokens are left in the iterator not
String	nextToken() The next token (word usually) in the string
void	setTokenPreProcessor(TokenPreProcess tokenPreProcessor) Set the token pre process

Method Detail

hasMoreTokens

boolean hasMoreTokens()

An iterator for tracking whether more tokens are left in the iterator not

Returns:

whether there is anymore tokens to iterate over

countTokens

int countTokens()

The number of tokens in the tokenizer

Returns:

the number of tokens

nextToken

String nextToken()

The next token (word usually) in the string

Returns:

the next token in the string if any

getTokens

List<String> getTokens()

Returns a list of all the tokens

Returns:

a list of all the tokens

setTokenPreProcessor

void setTokenPreProcessor(TokenPreProcess tokenPreProcessor)

Set the token pre process

Parameters:

tokenPreProcessor - the token pre processor to set