Package org.jruby.ext.ripper

Class RubyLexer

java.lang.Object

org.jruby.lexer.LexingCommon

org.jruby.ext.ripper.RubyLexer

public class RubyLexer
extends LexingCommon

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static enum	RubyLexer.Keyword

Field Summary

Fields

Modifier and Type	Field	Description
protected ByteList	delayed

Fields inherited from class org.jruby.lexer.LexingCommon

__end__seen, AMPERSAND, AMPERSAND_AMPERSAND, AMPERSAND_DOT, AND_KEYWORD, ASCII8BIT_ENCODING, BACKSLASH, BACKTICK, BANG, BANG_EQ, BANG_TILDE, BEGIN_DOC_MARKER, braceNest, CALL, CARET, cmdArgumentState, CODING, COLON, COLON_COLON, COMMA, commandStart, conditionState, DOLLAR_BANG, DOLLAR_DOT, DOLLAR_UNDERSCORE, DOT, DOT_DOT, DOT_DOT_DOT, end, END_DOC_MARKER, END_MARKER, EOF, eofp, EQ, EQ_EQ, EQ_EQ_EQ, EQ_GT, EQ_TILDE, EXPR_ARG, EXPR_ARG_ANY, EXPR_BEG, EXPR_BEG_ANY, EXPR_CLASS, EXPR_CMDARG, EXPR_DOT, EXPR_END, EXPR_END_ANY, EXPR_ENDARG, EXPR_ENDFN, EXPR_FITEM, EXPR_FNAME, EXPR_LABEL, EXPR_LABELED, EXPR_MID, EXPR_VALUE, GT, GT_EQ, GT_GT, has_shebang, heredoc_end, heredoc_indent, heredoc_line_indent, KWNOREST, last_cr_line, last_state, LBRACKET, LBRACKET_RBRACKET, LBRACKET_RBRACKET_EQ, LCURLY, lex_lastline, lex_nextline, lex_p, lex_pbeg, lex_pend, lex_state, lexb, line_count, line_offset, LT, LT_EQ, LT_EQ_RT, LT_LT, MINUS, MINUS_AT, MINUS_GT, NIL, OR, OR_KEYWORD, OR_OR, parenNest, PERCENT, PLUS, PLUS_AT, Q, QQ, QUESTION, RBRACKET, RCURLY, RPAREN, ruby_sourceline, SEMICOLON, SLASH, src, STAR, STAR_STAR, start, str_dquote, str_dsym, str_dword, STR_FUNC_ESCAPE, STR_FUNC_EXPAND, STR_FUNC_LABEL, STR_FUNC_LIST, STR_FUNC_QWORDS, STR_FUNC_REGEXP, STR_FUNC_SYMBOL, STR_FUNC_TERM, str_label, str_regexp, str_squote, str_ssym, str_sword, str_xquote, SUFFIX_ALL, SUFFIX_I, SUFFIX_R, TAB_WIDTH, TILDE, token, tokenSeen, tokline, tokp, USASCII_ENCODING, UTF8_ENCODING, yaccValue

Constructor Summary

Constructors

Constructor	Description
RubyLexer(RipperParserBase parser, LexerSource src)
RubyLexer(RipperParserBase parser, LexerSource src, IRubyWarnings _warnings)

Method Summary

Modifier and Type	Method	Description
void	addDelayedToken(int tok, int end)
protected void	ambiguousOperator(String op, String syn)
void	compile_error(String message)
IRubyObject	createStr(ByteList buffer, int flags)
void	dispatchDelayedToken(int token)
void	dispatchHeredocEnd()
void	dispatchIgnoredScanEvent(int token)
void	dispatchScanEvent(int token)
protected void	flush_string_content(org.jcodings.Encoding encoding)
String	getIdent()
static RubyLexer.Keyword	getKeyword(String str)
Ruby	getRuntime()
StrTerm	getStrTerm()
boolean	hasScanEvent()
boolean	hasStarted()	Has lexing started yet?
void	heredoc_restore(HeredocTerm here)
ByteList	id()
boolean	isVerbose()
protected void	mismatchedRegexpEncodingError(org.jcodings.Encoding optionEncoding, org.jcodings.Encoding encoding)
int	nextc()
int	nextToken()
protected boolean	onMagicComment(String name, ByteList value)
void	parse_error(String message)
protected RegexpOptions	parseRegexpFlags()
protected void	set_yylval_id(ByteList id)
protected void	set_yylval_name(ByteList name)
protected void	set_yylval_noname()
protected void	set_yylval_val(ByteList name)
protected void	setCompileOptionFlag(String name, ByteList value)
protected void	setEncoding(ByteList name)
void	setParser(RipperParserBase parserSupport)	Parse must pass its support object for some check at bottom of yylex().
void	setStrTerm(StrTerm strterm)
protected void	setTokenInfo(String name, ByteList value)
protected int	src_line()
ByteList	tokenByteList()
int	tokenize_ident(int result)
void	warn(String message)
void	warning(String fmt)
void	warning(String fmt, String arg)

Methods inherited from class org.jruby.lexer.LexingCommon

checkRegexpFragment, checkRegexpSyntax, column, comment_at_top, compile_error_pos, createAsEncodedString, createTokenByteList, createTokenByteList, createTokenString, createTokenString, dedent_string, flush, getBraceNest, getCmdArgumentState, getConditionState, getCurrentArg, getCurrentLine, getEncoding, getFile, getHeredocIndent, getHeredocLineIndent, getLeftParenBegin, getLexContext, getLineOffset, getParenNest, getRubySourceline, getState, getTokenCR, incrementParenNest, IS_AFTER_OPERATOR, IS_ARG, IS_BEG, IS_END, IS_LABEL_POSSIBLE, IS_LABEL_SUFFIX, IS_lex_state, IS_lex_state_all, IS_SPCARG, isASCII, isASCII, isEndSeen, isGlobalCharPunct, isHexChar, isIdentifierChar, isLambdaBeginning, isLookingAtEOL, isNext_identchar, isOctChar, isSpace, ISSPACE, lex_goto_eol, lineno, magicCommentEncoding, magicCommentMarker, newtok, numberLiteralSuffix, optionsEncodingChar, p, parser_magic_comment, parser_prepare, parseRegexpFlags, peek, peek, peekVariableName, precise_mbclen, printState, pushback, readEscape, readUTF8EscapeIntoBuffer, readUTFEscape, readUTFEscapeRegexpLiteral, reset, resetStacks, scanHex, scanHexLiteral, scanOct, set_file_encoding, setBraceNest, setCurrentArg, setCurrentEncoding, setEncoding, setHeredocIndent, setHeredocLineIndent, setLeftParenBegin, setLexContext, setRegexpEncoding, setRubySourceline, setSource, setState, setValue, strncmp, tokAdd, tokadd_ident, tokadd_mbchar, tokadd_mbchar, tokaddmbc, tokCopy, token, tokHex, update_heredoc_indent, updateStartPosition, updateTokenPosition, validateFormalIdentifier, validateFormalIdentifier, value, warn_balanced, was_bol, whole_match_p

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

delayed

protected ByteList delayed

Constructor Details

RubyLexer

public RubyLexer(RipperParserBase parser,
 LexerSource src)

RubyLexer

public RubyLexer(RipperParserBase parser,
 LexerSource src,
 IRubyWarnings _warnings)

Method Details

ambiguousOperator

protected void ambiguousOperator(String op,
 String syn)

Specified by:

ambiguousOperator in class LexingCommon

onMagicComment

protected boolean onMagicComment(String name,
 ByteList value)

Overrides:

onMagicComment in class LexingCommon

src_line

protected int src_line()

isVerbose

public boolean isVerbose()

warn

public void warn(String message)

warning

public void warning(String fmt)

warning

public void warning(String fmt,
 String arg)

getKeyword

public static RubyLexer.Keyword getKeyword(String str)

hasStarted

public boolean hasStarted()

Has lexing started yet?

flush_string_content

protected void flush_string_content(org.jcodings.Encoding encoding)

addDelayedToken

public void addDelayedToken(int tok,
 int end)

nextc

public int nextc()

Specified by:

nextc in class LexingCommon

dispatchHeredocEnd

public void dispatchHeredocEnd()

compile_error

public void compile_error(String message)

Specified by:

compile_error in class LexingCommon

parse_error

public void parse_error(String message)

Specified by:

parse_error in class LexingCommon

set_yylval_id

protected void set_yylval_id(ByteList id)

Overrides:

set_yylval_id in class LexingCommon

id

public ByteList id()

Overrides:

id in class LexingCommon

set_yylval_noname

protected void set_yylval_noname()

set_yylval_name

protected void set_yylval_name(ByteList name)

Overrides:

set_yylval_name in class LexingCommon

set_yylval_val

protected void set_yylval_val(ByteList name)

tokenize_ident

public int tokenize_ident(int result)

Specified by:

tokenize_ident in class LexingCommon

heredoc_restore

public void heredoc_restore(HeredocTerm here)

nextToken

public int nextToken()
              throws IOException

Throws:

IOException

getIdent

public String getIdent()

getRuntime

public Ruby getRuntime()

setParser

public void setParser(RipperParserBase parserSupport)

Parse must pass its support object for some check at bottom of yylex(). Ruby does it this way as well (i.e. a little parsing logic in the lexer).

Parameters:

parserSupport -

setCompileOptionFlag

protected void setCompileOptionFlag(String name,
 ByteList value)

Specified by:

setCompileOptionFlag in class LexingCommon

parseRegexpFlags

protected RegexpOptions parseRegexpFlags()
                                  throws IOException

Specified by:

parseRegexpFlags in class LexingCommon

Throws:

IOException

mismatchedRegexpEncodingError

protected void mismatchedRegexpEncodingError(org.jcodings.Encoding optionEncoding,
 org.jcodings.Encoding encoding)

Specified by:

mismatchedRegexpEncodingError in class LexingCommon

setTokenInfo

protected void setTokenInfo(String name,
 ByteList value)

Specified by:

setTokenInfo in class LexingCommon

setEncoding

protected void setEncoding(ByteList name)

Specified by:

setEncoding in class LexingCommon

getStrTerm

public StrTerm getStrTerm()

setStrTerm

public void setStrTerm(StrTerm strterm)

createStr

public IRubyObject createStr(ByteList buffer,
 int flags)

tokenByteList

public ByteList tokenByteList()

hasScanEvent

public boolean hasScanEvent()

dispatchDelayedToken

public void dispatchDelayedToken(int token)

dispatchIgnoredScanEvent

public void dispatchIgnoredScanEvent(int token)

dispatchScanEvent

public void dispatchScanEvent(int token)