org.python.antlr

Class PythonTokenSource

java.lang.Object

org.python.antlr.PythonTokenSource

All Implemented Interfaces:

org.antlr.runtime.TokenSource

public class PythonTokenSource
extends java.lang.Object
implements org.antlr.runtime.TokenSource

Python does not explicitly provide begin and end nesting signals. Rather, the indentation level indicates when you begin and end. This is an interesting lexical problem because multiple DEDENT tokens should be sent to the parser sometimes without a corresponding input symbol! Consider the following example:

 a=1
 if a>1:
     print a
 b=3

Here the "b" token on the left edge signals that a DEDENT is needed after the "print a \n" and before the "b". The sequence should be

 ... 1 COLON NEWLINE INDENT PRINT a NEWLINE DEDENT b ASSIGN 3 ...

For more examples, see the big comment at the bottom of this file. This TokenStream normally just passes tokens through to the parser. Upon NEWLINE token from the lexer, however, an INDENT or DEDENT token may need to be sent to the parser. The NEWLINE is the trigger for this class to do it's job. NEWLINE is saved and then the first token of the next line is examined. If non-leading-whitespace token, then check against stack for indent vs dedent. If LEADING_WS, then the column of the next non-whitespace token will dictate indent vs dedent. The column of the next real token is number of spaces in the LEADING_WS token + 1 (to move past the whitespace). The lexer grammar must set the text of the LEADING_WS token to be the proper number of spaces (and do tab conversion etc...). A stack of column numbers is tracked and used to detect changes in indent level from one token to the next. A queue of tokens is built up to hold multiple DEDENT tokens that are generated. Before asking the lexer for another token via nextToken(), the queue is flushed first one token at a time. Terence Parr and Loring Craymer February 2004

Field Summary

Fields

Modifier and Type	Field and Description
static int	FIRST_CHAR_POSITION
static int	MAX_INDENTS

Constructor Summary

Constructors

Constructor and Description
PythonTokenSource(org.antlr.runtime.CommonTokenStream stream, java.lang.String filename)
PythonTokenSource(org.antlr.runtime.CommonTokenStream stream, java.lang.String filename, boolean single)
PythonTokenSource(org.python.antlr.PythonLexer lexer)

Method Summary

Modifier and Type	Method and Description
protected int	findPreviousIndent(int i, org.antlr.runtime.Token t) Return the index on stack of previous indent level == i else -1
java.lang.String	getSourceName()
protected void	insertImaginaryIndentDedentTokens()
org.antlr.runtime.Token	nextToken() From http://www.python.org/doc/2.2.3/ref/indentation.html "Before the first line of the file is read, a single zero is pushed on the stack; this will never be popped off again.
protected int	peek()
protected int	pop()
protected void	push(int i)
java.lang.String	stackString()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAX_INDENTS

public static final int MAX_INDENTS

See Also:

Constant Field Values

FIRST_CHAR_POSITION

public static final int FIRST_CHAR_POSITION

See Also:

Constant Field Values

Constructor Detail

PythonTokenSource

public PythonTokenSource(org.python.antlr.PythonLexer lexer)

PythonTokenSource

public PythonTokenSource(org.antlr.runtime.CommonTokenStream stream,
                         java.lang.String filename)

PythonTokenSource

public PythonTokenSource(org.antlr.runtime.CommonTokenStream stream,
                         java.lang.String filename,
                         boolean single)

Method Detail

nextToken

public org.antlr.runtime.Token nextToken()

From http://www.python.org/doc/2.2.3/ref/indentation.html "Before the first line of the file is read, a single zero is pushed on the stack; this will never be popped off again. The numbers pushed on the stack will always be strictly increasing from bottom to top. At the beginning of each logical line, the line's indentation level is compared to the top of the stack. If it is equal, nothing happens. If it is larger, it is pushed on the stack, and one INDENT token is generated. If it is smaller, it must be one of the numbers occurring on the stack; all numbers on the stack that are larger are popped off, and for each number popped off a DEDENT token is generated. At the end of the file, a DEDENT token is generated for each number remaining on the stack that is larger than zero." I use char position in line 0..n-1 instead. The DEDENTS possibly needed at EOF are gracefully handled by forcing EOF to have char pos 0 even though with UNIX it's hard to get EOF at a non left edge.

Specified by:

nextToken in interface org.antlr.runtime.TokenSource

insertImaginaryIndentDedentTokens

protected void insertImaginaryIndentDedentTokens()

push

protected void push(int i)

pop

protected int pop()

peek

protected int peek()

findPreviousIndent

protected int findPreviousIndent(int i,
                                 org.antlr.runtime.Token t)

Return the index on stack of previous indent level == i else -1

stackString

public java.lang.String stackString()

getSourceName

public java.lang.String getSourceName()

Specified by:

getSourceName in interface org.antlr.runtime.TokenSource