Package org.deeplearning4j.text.tokenization.tokenizer

Class DefaultStreamTokenizer

java.lang.Object

org.deeplearning4j.text.tokenization.tokenizer.DefaultStreamTokenizer

All Implemented Interfaces:

Tokenizer

public class DefaultStreamTokenizer
extends Object
implements Tokenizer

Tokenizer based on the StreamTokenizer

Author:

Adam Gibson

Field Summary

Fields

Modifier and Type	Field	Description
protected static org.slf4j.Logger	log

Constructor Summary

Constructors

Constructor	Description
DefaultStreamTokenizer(InputStream is)

Method Summary

Modifier and Type	Method	Description
int	countTokens()	Returns number of tokens PLEASE NOTE: this method effectively preloads all tokens.
List<String>	getTokens()	Returns all tokens as list of Strings
boolean	hasMoreTokens()	Checks, if any prebuffered tokens left, otherswise checks underlying stream
String	nextToken()	This method returns next token from prebuffered list of tokens or underlying InputStream
void	setTokenPreProcessor(TokenPreProcess tokenPreProcessor)	Set the token pre process

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

log

protected static final org.slf4j.Logger log

Constructor Detail

DefaultStreamTokenizer

public DefaultStreamTokenizer(InputStream is)

Method Detail

hasMoreTokens

public boolean hasMoreTokens()

Checks, if any prebuffered tokens left, otherswise checks underlying stream

Specified by:

hasMoreTokens in interface Tokenizer

Returns:

countTokens

public int countTokens()

Returns number of tokens PLEASE NOTE: this method effectively preloads all tokens. So use it with caution, since on large streams it will consume big amount of memory

Specified by:

countTokens in interface Tokenizer

Returns:

nextToken

public String nextToken()

This method returns next token from prebuffered list of tokens or underlying InputStream

Specified by:

nextToken in interface Tokenizer

Returns:

next token as String

getTokens

public List<String> getTokens()

Returns all tokens as list of Strings

Specified by:

getTokens in interface Tokenizer

Returns:

List of tokens

setTokenPreProcessor

public void setTokenPreProcessor(TokenPreProcess tokenPreProcessor)

Description copied from interface: Tokenizer

Set the token pre process

Specified by:

setTokenPreProcessor in interface Tokenizer

Parameters:

tokenPreProcessor - the token pre processor to set