org.jruby.lexer.yacc

Class RubyYaccLexer

java.lang.Object

org.jruby.lexer.yacc.RubyYaccLexer

public class RubyYaccLexer
extends Object

This is a port of the MRI lexer to Java it is compatible to Ruby 1.8.1.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	RubyYaccLexer.Keyword
static class	RubyYaccLexer.LexState

Field Summary

Fields

Modifier and Type	Field and Description
static org.jcodings.Encoding	ASCII8BIT_ENCODING
boolean	commandStart
static org.jcodings.Encoding	USASCII_ENCODING
static org.jcodings.Encoding	UTF8_ENCODING

Constructor Summary

Constructors

Constructor and Description
RubyYaccLexer()
RubyYaccLexer(boolean isOneEight)

Method Summary

Methods

Modifier and Type	Method and Description
StrNode	createStrNode(ISourcePosition position, org.jruby.util.ByteList buffer, int flags)
StackState	getCmdArgumentState()
StackState	getConditionState()
String	getCurrentLine()
org.jcodings.Encoding	getEncoding()
static RubyYaccLexer.Keyword	getKeyword(String str)
int	getLeftParenBegin()
ISourcePosition	getPosition()
ISourcePosition	getPosition(ISourcePosition startPosition) Get position information for Token/Node that follows node represented by startPosition and current lexer location.
StrTerm	getStrTerm()
StringBuilder	getTokenBuffer()
protected void	handleFileEncodingComment(org.jruby.util.ByteList encodingLine)
int	incrementParenNest()
boolean	isGlobalCharPunct(int c)
boolean	isIdentifierChar(int c) This is a valid character for an identifier?
protected boolean	isMultiByteChar(int c) Is this a multibyte character from a multibyte encoding?
boolean	isOneEight()
int	nextToken()
protected boolean	parseMagicComment(org.jruby.util.ByteList magicLine)
protected int	readComment() Read a comment up to end of line.
int	readEscape()
int	readUTFEscape(org.jruby.util.ByteList buffer, boolean stringLiteral, boolean symbolLiteral)
void	readUTFEscapeRegexpLiteral(org.jruby.util.ByteList buffer)
void	reset()
void	resetStacks()
void	setEncoding(org.jcodings.Encoding encoding)
void	setLeftParenBegin(int value)
void	setParserSupport(ParserSupport parserSupport) Parse must pass its support object for some check at bottom of yylex().
void	setSource(LexerSource source) Allow the parser to set the source for its lexer.
void	setState(RubyYaccLexer.LexState state)
void	setStrTerm(StrTerm strterm)
void	setValue(Object yaccValue)
void	setWarnings(IRubyWarnings warnings)
int	token() Last token read from the lexer at the end of a call to yylex()
int	tokenAddMBC(int codepoint, org.jruby.util.ByteList buffer)
void	tokenAddMBCFromSrc(int c, org.jruby.util.ByteList buffer)
Object	value() Value of last token (if it is a token which has a value).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

UTF8_ENCODING

public static final org.jcodings.Encoding UTF8_ENCODING

USASCII_ENCODING

public static final org.jcodings.Encoding USASCII_ENCODING

ASCII8BIT_ENCODING

public static final org.jcodings.Encoding ASCII8BIT_ENCODING

commandStart

public boolean commandStart

Constructor Detail

RubyYaccLexer

public RubyYaccLexer()

RubyYaccLexer

public RubyYaccLexer(boolean isOneEight)

Method Detail

getEncoding

public org.jcodings.Encoding getEncoding()

getKeyword

public static RubyYaccLexer.Keyword getKeyword(String str)

incrementParenNest

public int incrementParenNest()

getLeftParenBegin

public int getLeftParenBegin()

setLeftParenBegin

public void setLeftParenBegin(int value)

reset

public final void reset()

nextToken

public int nextToken()
              throws IOException

Throws:

IOException

token

public int token()

Last token read from the lexer at the end of a call to yylex()

Returns:

last token read

getTokenBuffer

public StringBuilder getTokenBuffer()

value

public Object value()

Value of last token (if it is a token which has a value).

Returns:

value of last value-laden token

getPosition

public ISourcePosition getPosition(ISourcePosition startPosition)

Get position information for Token/Node that follows node represented by startPosition and current lexer location.

Parameters:

startPosition - previous node/token

inclusive - include previous node into position information of current node

Returns:

a new position

getPosition

public ISourcePosition getPosition()

getCurrentLine

public String getCurrentLine()

setParserSupport

public void setParserSupport(ParserSupport parserSupport)

Parse must pass its support object for some check at bottom of yylex(). Ruby does it this way as well (i.e. a little parsing logic in the lexer).

Parameters:

parserSupport -

setEncoding

public void setEncoding(org.jcodings.Encoding encoding)

setSource

public void setSource(LexerSource source)

Allow the parser to set the source for its lexer.

Parameters:

source - where the lexer gets raw data

getStrTerm

public StrTerm getStrTerm()

setStrTerm

public void setStrTerm(StrTerm strterm)

resetStacks

public void resetStacks()

setWarnings

public void setWarnings(IRubyWarnings warnings)

setState

public void setState(RubyYaccLexer.LexState state)

getCmdArgumentState

public StackState getCmdArgumentState()

isOneEight

public boolean isOneEight()

getConditionState

public StackState getConditionState()

setValue

public void setValue(Object yaccValue)

isIdentifierChar

public boolean isIdentifierChar(int c)

This is a valid character for an identifier?

Parameters:

c - is character to be compared

Returns:

whether c is an identifier or not mri: is_identchar

isMultiByteChar

protected boolean isMultiByteChar(int c)

Is this a multibyte character from a multibyte encoding?

Parameters:

c -

Returns:

whether c is an multibyte char or not

createStrNode

public StrNode createStrNode(ISourcePosition position,
                    org.jruby.util.ByteList buffer,
                    int flags)

parseMagicComment

protected boolean parseMagicComment(org.jruby.util.ByteList magicLine)
                             throws IOException

Throws:

IOException

handleFileEncodingComment

protected void handleFileEncodingComment(org.jruby.util.ByteList encodingLine)
                                  throws IOException

Throws:

IOException

readComment

protected int readComment()
                   throws IOException

Read a comment up to end of line.

Returns:

something or eof value

Throws:

IOException

isGlobalCharPunct

public boolean isGlobalCharPunct(int c)

readUTFEscapeRegexpLiteral

public void readUTFEscapeRegexpLiteral(org.jruby.util.ByteList buffer)
                                throws IOException

Throws:

IOException

tokenAddMBC

public int tokenAddMBC(int codepoint,
              org.jruby.util.ByteList buffer)

tokenAddMBCFromSrc

public void tokenAddMBCFromSrc(int c,
                      org.jruby.util.ByteList buffer)
                        throws IOException

Throws:

IOException

readUTFEscape

public int readUTFEscape(org.jruby.util.ByteList buffer,
                boolean stringLiteral,
                boolean symbolLiteral)
                  throws IOException

Throws:

IOException

readEscape

public int readEscape()
               throws IOException

Throws:

IOException