LexingCommon (JRuby Core 9.1.2.0 API)

java.lang.Object
- org.jruby.lexer.LexingCommon

Direct Known Subclasses:

RipperLexer, RubyLexer
```
public abstract class LexingCommon
extends Object
```
Code and constants common to both ripper and main parser.

Field Summary

Fields
Modifier and Type	Field and Description
`protected boolean`	`__end__seen`
`static org.jcodings.Encoding`	`ASCII8BIT_ENCODING`
`static org.jruby.util.ByteList`	`BEGIN_DOC_MARKER`
`protected int`	`braceNest`
`protected StackState`	`cmdArgumentState`
`static org.jruby.util.ByteList`	`CODING`
`boolean`	`commandStart`
`protected StackState`	`conditionState`
`static org.jruby.util.ByteList`	`END_DOC_MARKER`
`static org.jruby.util.ByteList`	`END_MARKER`
`static int`	`EOF`
`boolean`	`eofp`
`static int`	`EXPR_ARG`
`static int`	`EXPR_ARG_ANY`
`static int`	`EXPR_BEG`
`static int`	`EXPR_BEG_ANY`
`static int`	`EXPR_CLASS`
`static int`	`EXPR_CMDARG`
`static int`	`EXPR_DOT`
`static int`	`EXPR_END`
`static int`	`EXPR_END_ANY`
`static int`	`EXPR_ENDARG`
`static int`	`EXPR_ENDFN`
`static int`	`EXPR_FNAME`
`static int`	`EXPR_LABEL`
`static int`	`EXPR_LABELED`
`static int`	`EXPR_MID`
`static int`	`EXPR_VALUE`
`protected boolean`	`has_shebang`
`protected int`	`heredoc_end`
`protected int`	`heredoc_indent`
`protected int`	`heredoc_line_indent`
`boolean`	`inKwarg`
`protected int`	`last_cr_line`
`protected int`	`last_state`
`org.jruby.util.ByteList`	`lex_lastline`
`protected org.jruby.util.ByteList`	`lex_nextline`
`int`	`lex_p`
`protected int`	`lex_pbeg`
`int`	`lex_pend`
`protected int`	`lex_state`
`org.jruby.util.ByteList`	`lexb`
`protected int`	`line_count`
`protected int`	`line_offset`
`static org.joni.Regex`	`magicRegexp`
`static String`	`magicString`
`protected int`	`parenNest`
`protected int`	`ruby_sourceline`
`protected LexerSource`	`src`
`static int`	`str_dquote`
`static int`	`str_dsym`
`static int`	`STR_FUNC_ESCAPE`
`static int`	`STR_FUNC_EXPAND`
`static int`	`STR_FUNC_INDENT`
`static int`	`STR_FUNC_LABEL`
`static int`	`STR_FUNC_QWORDS`
`static int`	`STR_FUNC_REGEXP`
`static int`	`STR_FUNC_SYMBOL`
`static int`	`str_label`
`static int`	`str_regexp`
`static int`	`str_squote`
`static int`	`str_ssym`
`static int`	`str_xquote`
`static int`	`SUFFIX_ALL`
`static int`	`SUFFIX_I`
`static int`	`SUFFIX_R`
`static int`	`TAB_WIDTH`
`protected int`	`token`
`protected boolean`	`tokenSeen`
`ISourcePosition`	`tokline`
`int`	`tokp`
`static org.jcodings.Encoding`	`USASCII_ENCODING`
`static org.jcodings.Encoding`	`UTF8_ENCODING`
`protected Object`	`yaccValue`

Constructor Summary

Constructors
Constructor and Description

LexingCommon(LexerSource src)

Constructors
Constructor and Description
`LexingCommon(LexerSource src)`

Method Summary

All Methods Static Methods Instance Methods Abstract Methods Concrete Methods
Modifier and Type	Method and Description
`protected abstract void`	`ambiguousOperator(String op, String syn)`
`int`	`column()`
`protected boolean`	`comment_at_top()`
`abstract void`	`compile_error(String message)`
`String`	`createAsEncodedString(byte[] bytes, int start, int length, org.jcodings.Encoding encoding)`
`org.jruby.util.ByteList`	`createTokenByteList()`
`String`	`createTokenString()`
`String`	`createTokenString(int start)`
`protected int`	`dedent_string(org.jruby.util.ByteList string, int width)`
`protected void`	`flush()`
`int`	`getBraceNest()`
`StackState`	`getCmdArgumentState()`
`StackState`	`getConditionState()`
`String`	`getCurrentArg()`
`String`	`getCurrentLine()`
`org.jcodings.Encoding`	`getEncoding()`
`String`	`getFile()`
`int`	`getHeredocIndent()`
`int`	`getLeftParenBegin()`
`int`	`getLineOffset()`
`ISourcePosition`	`getPosition()`
`int`	`getState()`
`int`	`getTokenCR()`
`int`	`incrementParenNest()`
`protected boolean`	`isAfterOperator()`
`protected boolean`	`isARG()`
`boolean`	`isASCII()`
`boolean`	`isASCII(int c)`
`protected boolean`	`isBEG()`
`protected boolean`	`isEND()`
`boolean`	`isEndSeen()`
`boolean`	`isGlobalCharPunct(int c)`
`static boolean`	`isHexChar(int c)`
`boolean`	`isIdentifierChar(int c)` This is a valid character for an identifier?
`protected boolean`	`isLabelPossible(boolean commandState)`
`protected boolean`	`isLabelSuffix()`
`static boolean`	`isLexState(int state, int mask)`
`protected boolean`	`isLexStateAll(int state, int mask)`
`protected boolean`	`isNext_identchar()`
`static boolean`	`isOctChar(int c)`
`protected boolean`	`isSpaceArg(int c, boolean spaceSeen)`
`void`	`lex_goto_eol()`
`int`	`lineno()`
`protected void`	`magicCommentEncoding(org.jruby.util.ByteList encoding)`
`static int`	`magicCommentMarker(org.jruby.util.ByteList str, int begin)`
`void`	`newtok(boolean unreadOnce)`
`abstract int`	`nextc()`
`protected int`	`numberLiteralSuffix(int mask)`
`int`	`p(int offset)`
`boolean`	`parseMagicComment(Ruby runtime, org.jruby.util.ByteList magicLine)`
`void`	`parser_prepare()`
`boolean`	`peek(int c)`
`protected boolean`	`peek(int c, int n)`
`int`	`precise_mbclen()`
`void`	`printState()`
`void`	`pushback(int c)`
`void`	`reset()`
`void`	`resetStacks()`
`protected char`	`scanOct(int count)`
`protected void`	`set_file_encoding(int str, int send)`
`void`	`setBraceNest(int nest)`
`protected abstract void`	`setCompileOptionFlag(String name, org.jruby.util.ByteList value)`
`void`	`setCurrentArg(String current_arg)`
`void`	`setCurrentEncoding(org.jcodings.Encoding encoding)`
`protected abstract void`	`setEncoding(org.jruby.util.ByteList name)`
`void`	`setEncoding(org.jcodings.Encoding encoding)`
`void`	`setHeredocIndent(int heredoc_indent)`
`void`	`setHeredocLineIndent(int heredoc_line_indent)`
`void`	`setLeftParenBegin(int value)`
`void`	`setSource(LexerSource source)` Allow the parser to set the source for its lexer.
`void`	`setState(int state)`
`protected abstract void`	`setTokenInfo(String name, org.jruby.util.ByteList value)`
`void`	`setValue(Object yaccValue)`
`protected boolean`	`strncmp(org.jruby.util.ByteList one, org.jruby.util.ByteList two, int length)`
`boolean`	`tokadd_ident(int c)`
`boolean`	`tokadd_mbchar(int first_byte)` This differs from MRI in a few ways.
`boolean`	`tokadd_mbchar(int first_byte, org.jruby.util.ByteList buffer)`
`void`	`tokAdd(int first_byte, org.jruby.util.ByteList buffer)`
`void`	`tokaddmbc(int codepoint, org.jruby.util.ByteList buffer)` This looks deceptively like tokadd_mbchar(int, ByteList) but it differs in that it uses the bytelists encoding and the first parameter is a full codepoint and not the first byte of a mbc sequence.
`void`	`tokCopy(int length, org.jruby.util.ByteList buffer)`
`int`	`token()` Last token read from the lexer at the end of a call to yylex()
`abstract int`	`tokenize_ident(int result)`
`boolean`	`update_heredoc_indent(int c)`
`void`	`validateFormalIdentifier(String identifier)`
`Object`	`value()` Value of last token (if it is a token which has a value).
`protected void`	`warn_balanced(int c, boolean spaceSeen, String op, String syn)`
`boolean`	`was_bol()`
`boolean`	`whole_match_p(org.jruby.util.ByteList eos, boolean indent)`

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

EXPR_BEG
```
public static final int EXPR_BEG
```
See Also:

Constant Field Values

EXPR_END
```
public static final int EXPR_END
```
See Also:

Constant Field Values

EXPR_ENDARG
```
public static final int EXPR_ENDARG
```
See Also:

Constant Field Values

EXPR_ENDFN
```
public static final int EXPR_ENDFN
```
See Also:

Constant Field Values

EXPR_ARG
```
public static final int EXPR_ARG
```
See Also:

Constant Field Values

EXPR_CMDARG
```
public static final int EXPR_CMDARG
```
See Also:

Constant Field Values

EXPR_MID
```
public static final int EXPR_MID
```
See Also:

Constant Field Values

EXPR_FNAME
```
public static final int EXPR_FNAME
```
See Also:

Constant Field Values

EXPR_DOT
```
public static final int EXPR_DOT
```
See Also:

Constant Field Values

EXPR_CLASS
```
public static final int EXPR_CLASS
```
See Also:

Constant Field Values

EXPR_LABEL
```
public static final int EXPR_LABEL
```
See Also:

Constant Field Values

EXPR_LABELED
```
public static final int EXPR_LABELED
```
See Also:

Constant Field Values

EXPR_VALUE
```
public static final int EXPR_VALUE
```
See Also:

Constant Field Values

EXPR_BEG_ANY
```
public static final int EXPR_BEG_ANY
```
See Also:

Constant Field Values

EXPR_ARG_ANY
```
public static final int EXPR_ARG_ANY
```
See Also:

Constant Field Values

EXPR_END_ANY
```
public static final int EXPR_END_ANY
```
See Also:

Constant Field Values

braceNest
```
protected int braceNest
```

commandStart
```
public boolean commandStart
```

conditionState
```
protected StackState conditionState
```

cmdArgumentState
```
protected StackState cmdArgumentState
```

__end__seen
```
protected boolean __end__seen
```

eofp
```
public boolean eofp
```

has_shebang
```
protected boolean has_shebang
```

heredoc_end
```
protected int heredoc_end
```

heredoc_indent
```
protected int heredoc_indent
```

heredoc_line_indent
```
protected int heredoc_line_indent
```

inKwarg
```
public boolean inKwarg
```

last_cr_line
```
protected int last_cr_line
```

last_state
```
protected int last_state
```

lexb
```
public org.jruby.util.ByteList lexb
```

lex_lastline

public org.jruby.util.ByteList lex_lastline

lex_nextline

protected org.jruby.util.ByteList lex_nextline

lex_p
```
public int lex_p
```

lex_pbeg
```
protected int lex_pbeg
```

lex_pend
```
public int lex_pend
```

lex_state
```
protected int lex_state
```

line_count
```
protected int line_count
```

line_offset
```
protected int line_offset
```

parenNest
```
protected int parenNest
```

ruby_sourceline
```
protected int ruby_sourceline
```

src
```
protected LexerSource src
```

token
```
protected int token
```

tokenSeen
```
protected boolean tokenSeen
```

tokline
```
public ISourcePosition tokline
```

tokp
```
public int tokp
```

yaccValue
```
protected Object yaccValue
```

TAB_WIDTH
```
public static final int TAB_WIDTH
```
See Also:

Constant Field Values

STR_FUNC_ESCAPE
```
public static final int STR_FUNC_ESCAPE
```
See Also:

Constant Field Values

STR_FUNC_EXPAND
```
public static final int STR_FUNC_EXPAND
```
See Also:

Constant Field Values

STR_FUNC_REGEXP
```
public static final int STR_FUNC_REGEXP
```
See Also:

Constant Field Values

STR_FUNC_QWORDS
```
public static final int STR_FUNC_QWORDS
```
See Also:

Constant Field Values

STR_FUNC_SYMBOL
```
public static final int STR_FUNC_SYMBOL
```
See Also:

Constant Field Values

STR_FUNC_INDENT
```
public static final int STR_FUNC_INDENT
```
See Also:

Constant Field Values

STR_FUNC_LABEL
```
public static final int STR_FUNC_LABEL
```
See Also:

Constant Field Values

str_label
```
public static final int str_label
```
See Also:

Constant Field Values

str_squote
```
public static final int str_squote
```
See Also:

Constant Field Values

str_dquote
```
public static final int str_dquote
```
See Also:

Constant Field Values

str_xquote
```
public static final int str_xquote
```
See Also:

Constant Field Values

str_regexp
```
public static final int str_regexp
```
See Also:

Constant Field Values

str_ssym
```
public static final int str_ssym
```
See Also:

Constant Field Values

str_dsym
```
public static final int str_dsym
```
See Also:

Constant Field Values

EOF
```
public static final int EOF
```
See Also:

Constant Field Values

END_MARKER

public static org.jruby.util.ByteList END_MARKER

BEGIN_DOC_MARKER

public static org.jruby.util.ByteList BEGIN_DOC_MARKER

END_DOC_MARKER

public static org.jruby.util.ByteList END_DOC_MARKER

CODING

public static org.jruby.util.ByteList CODING

UTF8_ENCODING

public static final org.jcodings.Encoding UTF8_ENCODING

USASCII_ENCODING

public static final org.jcodings.Encoding USASCII_ENCODING

ASCII8BIT_ENCODING

public static final org.jcodings.Encoding ASCII8BIT_ENCODING

SUFFIX_R
```
public static final int SUFFIX_R
```
See Also:

Constant Field Values

SUFFIX_I
```
public static final int SUFFIX_I
```
See Also:

Constant Field Values

SUFFIX_ALL
```
public static final int SUFFIX_ALL
```
See Also:

Constant Field Values

magicString
```
public static final String magicString
```
See Also:

Constant Field Values

magicRegexp

public static final org.joni.Regex magicRegexp

Constructor Detail
- LexingCommon
```
public LexingCommon(LexerSource src)
```

Method Detail

column
```
public int column()
```

comment_at_top
```
protected boolean comment_at_top()
```

createTokenByteList

public org.jruby.util.ByteList createTokenByteList()

createTokenString

public String createTokenString(int start)

createAsEncodedString

public String createAsEncodedString(byte[] bytes,
                                    int start,
                                    int length,
                                    org.jcodings.Encoding encoding)

createTokenString
```
public String createTokenString()
```

dedent_string

protected int dedent_string(org.jruby.util.ByteList string,
                            int width)

flush
```
protected void flush()
```

getBraceNest
```
public int getBraceNest()
```

getCmdArgumentState

public StackState getCmdArgumentState()

getConditionState
```
public StackState getConditionState()
```

getCurrentArg
```
public String getCurrentArg()
```

getCurrentLine
```
public String getCurrentLine()
```

getEncoding

public org.jcodings.Encoding getEncoding()

getFile
```
public String getFile()
```

getHeredocIndent
```
public int getHeredocIndent()
```

getLeftParenBegin
```
public int getLeftParenBegin()
```

getPosition
```
public ISourcePosition getPosition()
```

getLineOffset
```
public int getLineOffset()
```

getState
```
public int getState()
```

getTokenCR
```
public int getTokenCR()
```

incrementParenNest
```
public int incrementParenNest()
```

isEndSeen
```
public boolean isEndSeen()
```

isASCII
```
public boolean isASCII()
```

isASCII
```
public boolean isASCII(int c)
```

isGlobalCharPunct

public boolean isGlobalCharPunct(int c)

isIdentifierChar
```
public boolean isIdentifierChar(int c)
```
This is a valid character for an identifier?

Parameters:

c - is character to be compared

Returns:

whether c is an identifier or not mri: is_identchar

lex_goto_eol
```
public void lex_goto_eol()
```

lineno
```
public int lineno()
```

magicCommentEncoding

protected void magicCommentEncoding(org.jruby.util.ByteList encoding)

newtok

public void newtok(boolean unreadOnce)

numberLiteralSuffix

protected int numberLiteralSuffix(int mask)
                           throws IOException

Throws:: IOException

parser_prepare
```
public void parser_prepare()
```

p
```
public int p(int offset)
```

peek
```
public boolean peek(int c)
```

peek

protected boolean peek(int c,
                       int n)

precise_mbclen
```
public int precise_mbclen()
```

printState
```
public void printState()
```

pushback
```
public void pushback(int c)
```

reset
```
public void reset()
```

resetStacks
```
public void resetStacks()
```

scanOct

protected char scanOct(int count)
                throws IOException

Throws:: IOException

setCurrentArg

public void setCurrentArg(String current_arg)

setCurrentEncoding

public void setCurrentEncoding(org.jcodings.Encoding encoding)

setEncoding

public void setEncoding(org.jcodings.Encoding encoding)

set_file_encoding

protected void set_file_encoding(int str,
                                 int send)

setHeredocLineIndent

public void setHeredocLineIndent(int heredoc_line_indent)

setHeredocIndent

public void setHeredocIndent(int heredoc_indent)

setBraceNest
```
public void setBraceNest(int nest)
```

setLeftParenBegin

public void setLeftParenBegin(int value)

setSource
```
public void setSource(LexerSource source)
```
Allow the parser to set the source for its lexer.

Parameters:

source - where the lexer gets raw data

setState
```
public void setState(int state)
```

setValue

public void setValue(Object yaccValue)

strncmp

protected boolean strncmp(org.jruby.util.ByteList one,
                          org.jruby.util.ByteList two,
                          int length)

tokAdd

public void tokAdd(int first_byte,
                   org.jruby.util.ByteList buffer)

tokCopy

public void tokCopy(int length,
                    org.jruby.util.ByteList buffer)

tokadd_ident
```
public boolean tokadd_ident(int c)
```

tokadd_mbchar
```
public boolean tokadd_mbchar(int first_byte)
```
This differs from MRI in a few ways. This version does not apply value to a separate token buffer. It is for use when we know we will not be omitting or including ant non-syntactical characters. Use tokadd_mbchar(int, ByteList) if the string differs from actual source. Secondly, this returns a boolean instead of the first byte passed. MRI only used the return value as a success/failure code to return EOF. Because this version does not use a separate token buffer we only just increment lex_p. When we reach end of the token it will just get the bytes directly from source directly.

tokadd_mbchar

public boolean tokadd_mbchar(int first_byte,
                             org.jruby.util.ByteList buffer)

tokaddmbc
```
public void tokaddmbc(int codepoint,
                      org.jruby.util.ByteList buffer)
```
This looks deceptively like tokadd_mbchar(int, ByteList) but it differs in that it uses the bytelists encoding and the first parameter is a full codepoint and not the first byte of a mbc sequence.

token
```
public int token()
```
Last token read from the lexer at the end of a call to yylex()

Returns:

last token read

update_heredoc_indent

public boolean update_heredoc_indent(int c)

validateFormalIdentifier

public void validateFormalIdentifier(String identifier)

value
```
public Object value()
```
Value of last token (if it is a token which has a value).

Returns:

value of last value-laden token

warn_balanced

protected void warn_balanced(int c,
                             boolean spaceSeen,
                             String op,
                             String syn)

was_bol
```
public boolean was_bol()
```

whole_match_p

public boolean whole_match_p(org.jruby.util.ByteList eos,
                             boolean indent)

ambiguousOperator

protected abstract void ambiguousOperator(String op,
                                          String syn)

compile_error

public abstract void compile_error(String message)

nextc
```
public abstract int nextc()
```

setCompileOptionFlag

protected abstract void setCompileOptionFlag(String name,
                                             org.jruby.util.ByteList value)

setEncoding

protected abstract void setEncoding(org.jruby.util.ByteList name)

setTokenInfo

protected abstract void setTokenInfo(String name,
                                     org.jruby.util.ByteList value)

tokenize_ident

public abstract int tokenize_ident(int result)

isHexChar
```
public static boolean isHexChar(int c)
```
Parameters:

c - the character to test

Returns:

true if character is a hex value (0-9a-f)

isLexState

public static boolean isLexState(int state,
                                 int mask)

isLexStateAll

protected boolean isLexStateAll(int state,
                                int mask)

isARG
```
protected boolean isARG()
```

isBEG
```
protected boolean isBEG()
```

isEND
```
protected boolean isEND()
```

isLabelPossible

protected boolean isLabelPossible(boolean commandState)

isLabelSuffix
```
protected boolean isLabelSuffix()
```

isAfterOperator
```
protected boolean isAfterOperator()
```

isNext_identchar

protected boolean isNext_identchar()
                            throws IOException

Throws:: IOException

isOctChar
```
public static boolean isOctChar(int c)
```
Parameters:

c - the character to test

Returns:

true if character is an octal value (0-7)

isSpaceArg

protected boolean isSpaceArg(int c,
                             boolean spaceSeen)

magicCommentMarker

public static int magicCommentMarker(org.jruby.util.ByteList str,
                                     int begin)

parseMagicComment

public boolean parseMagicComment(Ruby runtime,
                                 org.jruby.util.ByteList magicLine)
                          throws IOException

Throws:: IOException

Class LexingCommon

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

EXPR_BEG

EXPR_END

EXPR_ENDARG

EXPR_ENDFN

EXPR_ARG

EXPR_CMDARG

EXPR_MID

EXPR_FNAME

EXPR_DOT

EXPR_CLASS

EXPR_LABEL

EXPR_LABELED

EXPR_VALUE

EXPR_BEG_ANY

EXPR_ARG_ANY

EXPR_END_ANY

braceNest

commandStart

conditionState

cmdArgumentState

__end__seen

eofp

has_shebang

heredoc_end

heredoc_indent

heredoc_line_indent

inKwarg

last_cr_line

last_state

lexb

lex_lastline

lex_nextline

lex_p

lex_pbeg

lex_pend

lex_state

line_count

line_offset

parenNest

ruby_sourceline

src

token

tokenSeen

tokline

tokp

yaccValue

TAB_WIDTH

STR_FUNC_ESCAPE

STR_FUNC_EXPAND

STR_FUNC_REGEXP

STR_FUNC_QWORDS

STR_FUNC_SYMBOL

STR_FUNC_INDENT

STR_FUNC_LABEL

str_label

str_squote

str_dquote

str_xquote

str_regexp

str_ssym

str_dsym

EOF

END_MARKER

BEGIN_DOC_MARKER

END_DOC_MARKER

CODING

UTF8_ENCODING

USASCII_ENCODING

ASCII8BIT_ENCODING

SUFFIX_R

SUFFIX_I

SUFFIX_ALL

magicString

magicRegexp

endseen