org.jruby.lexer.yacc

Class RubyLexer

java.lang.Object

org.jruby.lexer.yacc.RubyLexer

public class RubyLexer
extends Object

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	RubyLexer.Keyword
static class	RubyLexer.LexState

Field Summary

Fields

Modifier and Type	Field and Description
static org.jcodings.Encoding	ASCII8BIT_ENCODING
boolean	commandStart
ISourcePosition	tokline
static org.jcodings.Encoding	USASCII_ENCODING
static org.jcodings.Encoding	UTF8_ENCODING

Constructor Summary

Constructors

Constructor and Description
RubyLexer()

Method Summary

Methods

Modifier and Type	Method and Description
StrNode	createStrNode(ISourcePosition position, org.jruby.util.ByteList buffer, int flags)
int	getBraceNest()
StackState	getCmdArgumentState()
StackState	getConditionState()
String	getCurrentLine()
org.jcodings.Encoding	getEncoding()
static RubyLexer.Keyword	getKeyword(String str)
int	getLeftParenBegin()
ISourcePosition	getPosition()
ISourcePosition	getPosition(ISourcePosition startPosition) Get position information for Token/Node that follows node represented by startPosition and current lexer location.
RubyLexer.LexState	getState()
StrTerm	getStrTerm()
StringBuilder	getTokenBuffer()
protected void	handleFileEncodingComment(org.jruby.util.ByteList encodingLine)
int	incrementParenNest()
boolean	isASCII(int c)
boolean	isGlobalCharPunct(int c)
boolean	isIdentifierChar(int c) This is a valid character for an identifier?
protected boolean	isMultiByteChar(int c) Is this a multibyte character from a multibyte encoding?
void	newtok()
int	nextToken()
protected boolean	parseMagicComment(org.jruby.util.ByteList magicLine)
protected int	readComment() Read a comment up to end of line.
int	readEscape()
int	readUTFEscape(org.jruby.util.ByteList buffer, boolean stringLiteral, boolean symbolLiteral)
void	readUTFEscapeRegexpLiteral(org.jruby.util.ByteList buffer)
void	reset()
void	resetStacks()
void	setBraceNest(int nest)
void	setEncoding(org.jcodings.Encoding encoding)
void	setLeftParenBegin(int value)
void	setParserSupport(ParserSupport parserSupport) Parse must pass its support object for some check at bottom of yylex().
void	setSource(LexerSource source) Allow the parser to set the source for its lexer.
void	setState(RubyLexer.LexState state)
void	setStrTerm(StrTerm strterm)
void	setValue(Object yaccValue)
void	setWarnings(IRubyWarnings warnings)
void	startOfToken()
int	token() Last token read from the lexer at the end of a call to yylex()
int	tokenAddMBC(int codepoint, org.jruby.util.ByteList buffer)
void	tokenAddMBCFromSrc(int c, org.jruby.util.ByteList buffer)
Object	value() Value of last token (if it is a token which has a value).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

UTF8_ENCODING

public static final org.jcodings.Encoding UTF8_ENCODING

USASCII_ENCODING

public static final org.jcodings.Encoding USASCII_ENCODING

ASCII8BIT_ENCODING

public static final org.jcodings.Encoding ASCII8BIT_ENCODING

tokline

public ISourcePosition tokline

commandStart

public boolean commandStart

Constructor Detail

RubyLexer

public RubyLexer()

Method Detail

getEncoding

public org.jcodings.Encoding getEncoding()

getKeyword

public static RubyLexer.Keyword getKeyword(String str)

startOfToken

public void startOfToken()

newtok

public void newtok()

incrementParenNest

public int incrementParenNest()

getBraceNest

public int getBraceNest()

setBraceNest

public void setBraceNest(int nest)

getLeftParenBegin

public int getLeftParenBegin()

setLeftParenBegin

public void setLeftParenBegin(int value)

reset

public final void reset()

nextToken

public int nextToken()
              throws IOException

Throws:

IOException

token

public int token()

Last token read from the lexer at the end of a call to yylex()

Returns:

last token read

getTokenBuffer

public StringBuilder getTokenBuffer()

value

public Object value()

Value of last token (if it is a token which has a value).

Returns:

value of last value-laden token

getPosition

public ISourcePosition getPosition(ISourcePosition startPosition)

Get position information for Token/Node that follows node represented by startPosition and current lexer location.

Parameters:

startPosition - previous node/token

Returns:

a new position

getPosition

public ISourcePosition getPosition()

getCurrentLine

public String getCurrentLine()

setParserSupport

public void setParserSupport(ParserSupport parserSupport)

Parse must pass its support object for some check at bottom of yylex(). Ruby does it this way as well (i.e. a little parsing logic in the lexer).

Parameters:

parserSupport -

setEncoding

public void setEncoding(org.jcodings.Encoding encoding)

setSource

public void setSource(LexerSource source)

Allow the parser to set the source for its lexer.

Parameters:

source - where the lexer gets raw data

getStrTerm

public StrTerm getStrTerm()

setStrTerm

public void setStrTerm(StrTerm strterm)

resetStacks

public void resetStacks()

setWarnings

public void setWarnings(IRubyWarnings warnings)

getState

public RubyLexer.LexState getState()

setState

public void setState(RubyLexer.LexState state)

getCmdArgumentState

public StackState getCmdArgumentState()

getConditionState

public StackState getConditionState()

setValue

public void setValue(Object yaccValue)

isIdentifierChar

public boolean isIdentifierChar(int c)

This is a valid character for an identifier?

Parameters:

c - is character to be compared

Returns:

whether c is an identifier or not mri: is_identchar

isASCII

public boolean isASCII(int c)

isMultiByteChar

protected boolean isMultiByteChar(int c)

Is this a multibyte character from a multibyte encoding?

Parameters:

c -

Returns:

whether c is an multibyte char or not

createStrNode

public StrNode createStrNode(ISourcePosition position,
                    org.jruby.util.ByteList buffer,
                    int flags)

parseMagicComment

protected boolean parseMagicComment(org.jruby.util.ByteList magicLine)
                             throws IOException

Throws:

IOException

handleFileEncodingComment

protected void handleFileEncodingComment(org.jruby.util.ByteList encodingLine)
                                  throws IOException

Throws:

IOException

readComment

protected int readComment()
                   throws IOException

Read a comment up to end of line.

Returns:

something or eof value

Throws:

IOException

isGlobalCharPunct

public boolean isGlobalCharPunct(int c)

readUTFEscapeRegexpLiteral

public void readUTFEscapeRegexpLiteral(org.jruby.util.ByteList buffer)
                                throws IOException

Throws:

IOException

tokenAddMBC

public int tokenAddMBC(int codepoint,
              org.jruby.util.ByteList buffer)

tokenAddMBCFromSrc

public void tokenAddMBCFromSrc(int c,
                      org.jruby.util.ByteList buffer)
                        throws IOException

Throws:

IOException

readUTFEscape

public int readUTFEscape(org.jruby.util.ByteList buffer,
                boolean stringLiteral,
                boolean symbolLiteral)
                  throws IOException

Throws:

IOException

readEscape

public int readEscape()
               throws IOException

Throws:

IOException