Package net.sourceforge.pmd.cpd.impl
Class TokenizerBase<T extends GenericToken<T>>
- java.lang.Object
-
- net.sourceforge.pmd.cpd.impl.TokenizerBase<T>
-
- All Implemented Interfaces:
Tokenizer
- Direct Known Subclasses:
AntlrTokenizer
,JavaCCTokenizer
public abstract class TokenizerBase<T extends GenericToken<T>> extends Object implements Tokenizer
Generic base class for aTokenizer
.
-
-
Constructor Summary
Constructors Constructor Description TokenizerBase()
-
Method Summary
All Methods Instance Methods Abstract Methods Concrete Methods Modifier and Type Method Description protected TokenManager<T>
filterTokenStream(TokenManager<T> tokenManager)
protected String
getImage(T token)
protected abstract TokenManager<T>
makeLexerImpl(TextDocument doc)
protected void
processToken(TokenFactory tokenEntries, T currentToken)
void
tokenize(TextDocument document, TokenFactory tokens)
Tokenize the source code and record tokens using the provided token factory.
-
-
-
Method Detail
-
makeLexerImpl
protected abstract TokenManager<T> makeLexerImpl(TextDocument doc) throws IOException
- Throws:
IOException
-
filterTokenStream
protected TokenManager<T> filterTokenStream(TokenManager<T> tokenManager)
-
processToken
protected void processToken(TokenFactory tokenEntries, T currentToken)
-
tokenize
public final void tokenize(TextDocument document, TokenFactory tokens) throws IOException
Description copied from interface:Tokenizer
Tokenize the source code and record tokens using the provided token factory.- Specified by:
tokenize
in interfaceTokenizer
- Throws:
IOException
-
-