org.deeplearning4j.text.tokenization.tokenizer

Interface TokenPreProcess

All Known Implementing Classes:

BertWordPiecePreProcessor, CommonPreprocessor, CompositePreProcessor, EndingPreProcessor, LowCasePreProcessor

public interface TokenPreProcess

Method Summary

Modifier and Type	Method and Description
String	preProcess(String token) Pre process a token

Method Detail

preProcess

String preProcess(String token)

Pre process a token

Parameters:

token - the token to pre process

Returns:

the preprocessed token