Package com.sun.tools.javac.parser

Class JavaTokenizer

java.lang.Object

com.sun.tools.javac.parser.JavaTokenizer

Direct Known Subclasses:

JavadocTokenizer

public class JavaTokenizer
extends Object

The lexical analyzer maps an input stream consisting of ASCII characters and Unicode escapes into a token sequence.

This is NOT part of any supported API. If you write code that depends on this, you do so at your own risk. This code and its internal interfaces are subject to change or deletion without notice.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
protected static class	JavaTokenizer.BasicComment<U extends UnicodeReader> Scan a documentation comment; determine if a deprecated tag is present.

Field Summary

Fields

Modifier and Type	Field and Description
protected int	errPos The position where a lexical error occurred;
protected ScannerFactory	fac
protected Name	name The token's name, set by nextToken().
protected int	radix The token's radix, set by nextToken().
protected UnicodeReader	reader The Unicode reader (low-level stream reader).
protected Tokens.TokenKind	tk The token kind, set by nextToken().

Constructor Summary

Constructors

Modifier	Constructor and Description
protected	JavaTokenizer(ScannerFactory fac, char[] buf, int inputLength)
protected	JavaTokenizer(ScannerFactory fac, CharBuffer buf) Create a scanner from the input array.
protected	JavaTokenizer(ScannerFactory fac, UnicodeReader reader)

Method Summary

Modifier and Type	Method and Description
int	errPos() Return the position where a lexical error occurred;
void	errPos(int pos) Set the position where a lexical error occurred;
Position.LineMap	getLineMap() Build a map for translating between line numbers and positions in the input.
protected void	lexError(int pos, String key, Object... args) Report an error at the given position using the provided arguments.
protected Tokens.Comment	processComment(int pos, int endPos, Tokens.Comment.CommentStyle style) Called when a complete comment has been scanned.
protected void	processLineTerminator(int pos, int endPos) Called when a line terminator has been processed.
protected void	processWhiteSpace(int pos, int endPos) Called when a complete whitespace run has been scanned.
Tokens.Token	readToken() Read token.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

tk

protected Tokens.TokenKind tk

The token kind, set by nextToken().

radix

protected int radix

The token's radix, set by nextToken().

name

protected Name name

The token's name, set by nextToken().

errPos

protected int errPos

The position where a lexical error occurred;

reader

protected UnicodeReader reader

The Unicode reader (low-level stream reader).

fac

protected ScannerFactory fac

Constructor Detail

JavaTokenizer

protected JavaTokenizer(ScannerFactory fac,
                        CharBuffer buf)

Create a scanner from the input array. This method might modify the array. To avoid copying the input array, ensure that inputLength < input.length or input[input.length -1] is a white space character.

Parameters:

fac - the factory which created this Scanner

buf - the input, might be modified Must be positive and less than or equal to input.length.

JavaTokenizer

protected JavaTokenizer(ScannerFactory fac,
                        char[] buf,
                        int inputLength)

JavaTokenizer

protected JavaTokenizer(ScannerFactory fac,
                        UnicodeReader reader)

Method Detail

lexError

protected void lexError(int pos,
                        String key,
                        Object... args)

Report an error at the given position using the provided arguments.

readToken

public Tokens.Token readToken()

Read token.

errPos

public int errPos()

Return the position where a lexical error occurred;

errPos

public void errPos(int pos)

Set the position where a lexical error occurred;

processComment

protected Tokens.Comment processComment(int pos,
                                        int endPos,
                                        Tokens.Comment.CommentStyle style)

Called when a complete comment has been scanned. pos and endPos will mark the comment boundary.

processWhiteSpace

protected void processWhiteSpace(int pos,
                                 int endPos)

Called when a complete whitespace run has been scanned. pos and endPos will mark the whitespace boundary.

processLineTerminator

protected void processLineTerminator(int pos,
                                     int endPos)

Called when a line terminator has been processed.

getLineMap

public Position.LineMap getLineMap()

Build a map for translating between line numbers and positions in the input.

Returns:

a LineMap