Package org.antlr.codegen
Class CodeGenerator
- java.lang.Object
-
- org.antlr.codegen.CodeGenerator
-
public class CodeGenerator extends java.lang.Object
ANTLR's code generator. Generate recognizers derived from grammars. Language independence achieved through the use of STGroup objects. All output strings are completely encapsulated in the group files such as Java.stg. Some computations are done that are unused by a particular language. This generator just computes and sets the values into the templates; the templates are free to use or not use the information. To make a new code generation target, define X.stg for language X by copying from existing Y.stg most closely releated to your language; e.g., to do CSharp.stg copy Java.stg. The template group file has a bunch of templates that are needed by the code generator. You can add a new target w/o even recompiling ANTLR itself. The language=X option in a grammar file dictates which templates get loaded/used. Some language like C need both parser files and header files. Java needs to have a separate file for the cyclic DFA as ANTLR generates bytecodes directly (which cannot be in the generated parser Java file). To facilitate this, cyclic can be in same file, but header, output must be searpate. recognizer is in outptufile.
-
-
Field Summary
Fields Modifier and Type Field Description ACyclicDFACodeGenerator
acyclicDFAGenerator
I have factored out the generation of acyclic DFAs to separate classprotected org.stringtemplate.v4.STGroup
baseTemplates
The basic output templates without AST or templates stuff; this will be the templates loaded for the language such as Java.stg *and* the Dbg stuff if turned on.static java.lang.String
classpathTemplateRootDirectoryName
protected boolean
debug
Generate debugging event method callsboolean
GENERATE_SWITCHES_WHEN_POSSIBLE
Grammar
grammar
Which grammar are we generating code for? Each generator is attached to a specific grammar.protected org.stringtemplate.v4.ST
headerFileST
protected java.lang.String
language
What language are we generating?static boolean
LAUNCH_ST_INSPECTOR
protected int
lineWidth
static int
MADSI_DEFAULT
static int
MAX_ACYCLIC_DFA_STATES_INLINE
static int
MAX_SWITCH_CASE_LABELS
static int
MIN_SWITCH_ALTS
static int
MSA_DEFAULT
static int
MSCL_DEFAULT
When generating SWITCH statements, some targets might need to limit the size (based upon the number of case labels).protected org.stringtemplate.v4.ST
outputFileST
protected boolean
profile
Track runtime parsing information about decisions etc...protected org.stringtemplate.v4.ST
recognizerST
Target
target
The target specifies how to write out files and do other language specific actions.protected org.stringtemplate.v4.STGroup
templates
Where are the templates this generator should use to generate code?protected Tool
tool
A reference to the ANTLR tool so we can learn about output directories and such.protected boolean
trace
Create a Tracer object and make the recognizer invoke this.protected int
uniqueLabelNumber
Used to create unique labelsstatic java.lang.String
VOCAB_FILE_EXTENSION
I have factored out the generation of cyclic DFAs to separate classprotected static java.lang.String
vocabFilePattern
-
Constructor Summary
Constructors Constructor Description CodeGenerator(Tool tool, Grammar grammar, java.lang.String language)
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description protected boolean
canGenerateSwitch(DFAState s)
You can generate a switch rather than if-then-else for a DFA state if there are no semantic predicates and the number of edge label values is small enough; e.g., don't generate a switch for a state containing an edge label such as 20..52330 (the resulting byte codes would overflow the method 65k limit probably).java.lang.String
createUniqueLabel(java.lang.String name)
Create a label to track a token / rule reference's result.void
generateLocalFOLLOW(GrammarAST referencedElementNode, java.lang.String referencedElementName, java.lang.String enclosingRuleName, int elementIndex)
Error recovery in ANTLR recognizers.org.stringtemplate.v4.ST
generateSpecialState(DFAState s)
A special state is huge (too big for state tables) or has a predicated edge.protected org.stringtemplate.v4.ST
genLabelExpr(org.stringtemplate.v4.STGroup templates, Transition edge, int k)
Generate an expression for traversing an edge.org.stringtemplate.v4.ST
genLookaheadDecision(org.stringtemplate.v4.ST recognizerST, DFA dfa)
Generate code that computes the predicted alt given a DFA.org.stringtemplate.v4.ST
genRecognizer()
Given the grammar to which we are attached, walk the AST associated with that grammar to create NFAs.protected org.stringtemplate.v4.ST
genSemanticPredicateExpr(org.stringtemplate.v4.STGroup templates, Transition edge)
org.stringtemplate.v4.ST
genSetExpr(org.stringtemplate.v4.STGroup templates, IntSet set, int k, boolean partOfDFA)
For intervals such as [3..3, 30..35], generate an expression that tests the lookahead similar to LA(1)==3 || (LA(1)>=30&&LA(1)<=35)protected void
genTokenTypeConstants(org.stringtemplate.v4.ST code)
Set attributes tokens and literals attributes in the incoming code template.protected void
genTokenTypeNames(org.stringtemplate.v4.ST code)
Generate a token names table that maps token type to a printable name: either the label like INT or the literal like "begin".protected org.stringtemplate.v4.ST
genTokenVocabOutput()
Generate a token vocab file with all the token names/types.org.stringtemplate.v4.STGroup
getBaseTemplates()
static java.util.List<java.lang.String>
getListOfArgumentsFromAction(java.lang.String actionText, int separatorChar)
static int
getListOfArgumentsFromAction(java.lang.String actionText, int start, int targetChar, int separatorChar, java.util.List<java.lang.String> args)
Given an arg action like [x, (*a).foo(21,33), 3.2+1, '\n', "a,oo\nick", {bl, "fdkj"eck}, ["cat\n,", x, 43]] convert to a list of arguments.java.lang.String
getRecognizerFileName(java.lang.String name, int type)
Generate TParser.java and TLexer.java from T.g if combined, else just use T.java as output regardless of type.org.stringtemplate.v4.ST
getRecognizerST()
org.stringtemplate.v4.STGroup
getTemplates()
java.lang.String
getTokenTypeAsTargetLabel(int ttype)
Get a meaningful name for a token type useful during code generation.java.lang.String
getVocabFileName()
What is the name of the vocab file generated for this grammar? Returns null if no .tokens file should be generated.void
issueInvalidAttributeError(java.lang.String x, java.lang.String y, Rule enclosingRule, org.antlr.runtime.Token actionToken, int outerAltNum)
void
issueInvalidAttributeError(java.lang.String x, Rule enclosingRule, org.antlr.runtime.Token actionToken, int outerAltNum)
void
issueInvalidScopeError(java.lang.String x, java.lang.String y, Rule enclosingRule, org.antlr.runtime.Token actionToken, int outerAltNum)
static Target
loadLanguageTarget(java.lang.String language)
void
loadTemplates(java.lang.String language)
load the main language.stg template group filevoid
setDebug(boolean debug)
void
setProfile(boolean profile)
void
setTrace(boolean trace)
java.util.List<? extends java.lang.Object>
translateAction(java.lang.String ruleName, GrammarAST actionTree)
protected void
translateActionAttributeReferences(java.util.Map<java.lang.String,java.util.Map<java.lang.String,java.lang.Object>> actions)
Actions may reference $x::y attributes, call translateAction on each action and replace that action in the Map.void
translateActionAttributeReferencesForSingleScope(Rule r, java.util.Map<java.lang.String,java.lang.Object> scopeActions)
Use for translating rule @init{...} actions that have no scopejava.util.List<org.stringtemplate.v4.ST>
translateArgAction(java.lang.String ruleName, GrammarAST actionTree)
Translate an action like [3,"foo",a[3]] and return a List of the translated actions.org.stringtemplate.v4.ST
translateTemplateConstructor(java.lang.String ruleName, int outerAltNum, org.antlr.runtime.Token actionToken, java.lang.String templateActionText)
Given a template constructor action like %foo(a={...}) in an action, translate it to the appropriate template constructor from the templateLib.protected void
verifyActionScopesOkForTarget(java.util.Map<java.lang.String,java.util.Map<java.lang.String,java.lang.Object>> actions)
Some targets will have some extra scopes like C++ may have '@headerfile:name {action}' or something.void
write(org.stringtemplate.v4.ST code, java.lang.String fileName)
-
-
-
Field Detail
-
MSCL_DEFAULT
public static final int MSCL_DEFAULT
When generating SWITCH statements, some targets might need to limit the size (based upon the number of case labels). Generally, this limit will be hit only for lexers where wildcard in a UNICODE vocabulary environment would generate a SWITCH with 65000 labels.- See Also:
- Constant Field Values
-
MAX_SWITCH_CASE_LABELS
public static int MAX_SWITCH_CASE_LABELS
-
MSA_DEFAULT
public static final int MSA_DEFAULT
- See Also:
- Constant Field Values
-
MIN_SWITCH_ALTS
public static int MIN_SWITCH_ALTS
-
GENERATE_SWITCHES_WHEN_POSSIBLE
public boolean GENERATE_SWITCHES_WHEN_POSSIBLE
-
LAUNCH_ST_INSPECTOR
public static boolean LAUNCH_ST_INSPECTOR
-
MADSI_DEFAULT
public static final int MADSI_DEFAULT
- See Also:
- Constant Field Values
-
MAX_ACYCLIC_DFA_STATES_INLINE
public static int MAX_ACYCLIC_DFA_STATES_INLINE
-
classpathTemplateRootDirectoryName
public static java.lang.String classpathTemplateRootDirectoryName
-
grammar
public Grammar grammar
Which grammar are we generating code for? Each generator is attached to a specific grammar.
-
language
protected java.lang.String language
What language are we generating?
-
target
public Target target
The target specifies how to write out files and do other language specific actions.
-
templates
protected org.stringtemplate.v4.STGroup templates
Where are the templates this generator should use to generate code?
-
baseTemplates
protected org.stringtemplate.v4.STGroup baseTemplates
The basic output templates without AST or templates stuff; this will be the templates loaded for the language such as Java.stg *and* the Dbg stuff if turned on. This is used for generating syntactic predicates.
-
recognizerST
protected org.stringtemplate.v4.ST recognizerST
-
outputFileST
protected org.stringtemplate.v4.ST outputFileST
-
headerFileST
protected org.stringtemplate.v4.ST headerFileST
-
uniqueLabelNumber
protected int uniqueLabelNumber
Used to create unique labels
-
tool
protected Tool tool
A reference to the ANTLR tool so we can learn about output directories and such.
-
debug
protected boolean debug
Generate debugging event method calls
-
trace
protected boolean trace
Create a Tracer object and make the recognizer invoke this.
-
profile
protected boolean profile
Track runtime parsing information about decisions etc... This requires the debugging event mechanism to work.
-
lineWidth
protected int lineWidth
-
acyclicDFAGenerator
public ACyclicDFACodeGenerator acyclicDFAGenerator
I have factored out the generation of acyclic DFAs to separate class
-
VOCAB_FILE_EXTENSION
public static final java.lang.String VOCAB_FILE_EXTENSION
I have factored out the generation of cyclic DFAs to separate class- See Also:
- Constant Field Values
-
vocabFilePattern
protected static final java.lang.String vocabFilePattern
- See Also:
- Constant Field Values
-
-
Method Detail
-
loadLanguageTarget
public static Target loadLanguageTarget(java.lang.String language)
-
loadTemplates
public void loadTemplates(java.lang.String language)
load the main language.stg template group file
-
genRecognizer
public org.stringtemplate.v4.ST genRecognizer()
Given the grammar to which we are attached, walk the AST associated with that grammar to create NFAs. Then create the DFAs for all decision points in the grammar by converting the NFAs to DFAs. Finally, walk the AST again to generate code. Either 1 or 2 files are written: recognizer: the main parser/lexer/treewalker item header file: language like C/C++ need extern definitions The target, such as JavaTarget, dictates which files get written.
-
verifyActionScopesOkForTarget
protected void verifyActionScopesOkForTarget(java.util.Map<java.lang.String,java.util.Map<java.lang.String,java.lang.Object>> actions)
Some targets will have some extra scopes like C++ may have '@headerfile:name {action}' or something. Make sure the target likes the scopes in action table.
-
translateActionAttributeReferences
protected void translateActionAttributeReferences(java.util.Map<java.lang.String,java.util.Map<java.lang.String,java.lang.Object>> actions)
Actions may reference $x::y attributes, call translateAction on each action and replace that action in the Map.
-
translateActionAttributeReferencesForSingleScope
public void translateActionAttributeReferencesForSingleScope(Rule r, java.util.Map<java.lang.String,java.lang.Object> scopeActions)
Use for translating rule @init{...} actions that have no scope
-
generateLocalFOLLOW
public void generateLocalFOLLOW(GrammarAST referencedElementNode, java.lang.String referencedElementName, java.lang.String enclosingRuleName, int elementIndex)
Error recovery in ANTLR recognizers. Based upon original ideas: Algorithms + Data Structures = Programs by Niklaus Wirth and A note on error recovery in recursive descent parsers: http://portal.acm.org/citation.cfm?id=947902.947905 Later, Josef Grosch had some good ideas: Efficient and Comfortable Error Recovery in Recursive Descent Parsers: ftp://www.cocolab.com/products/cocktail/doca4.ps/ell.ps.zip Like Grosch I implemented local FOLLOW sets that are combined at run-time upon error to avoid parsing overhead.
-
genLookaheadDecision
public org.stringtemplate.v4.ST genLookaheadDecision(org.stringtemplate.v4.ST recognizerST, DFA dfa)
Generate code that computes the predicted alt given a DFA. The recognizerST can be either the main generated recognizerTemplate for storage in the main parser file or a separate file. It's up to the code that ultimately invokes the codegen.g grammar rule. Regardless, the output file and header file get a copy of the DFAs.
-
generateSpecialState
public org.stringtemplate.v4.ST generateSpecialState(DFAState s)
A special state is huge (too big for state tables) or has a predicated edge. Generate a simple if-then-else. Cannot be an accept state as they have no emanating edges. Don't worry about switch vs if-then-else because if you get here, the state is super complicated and needs an if-then-else. This is used by the new DFA scheme created June 2006.
-
genLabelExpr
protected org.stringtemplate.v4.ST genLabelExpr(org.stringtemplate.v4.STGroup templates, Transition edge, int k)
Generate an expression for traversing an edge.
-
genSemanticPredicateExpr
protected org.stringtemplate.v4.ST genSemanticPredicateExpr(org.stringtemplate.v4.STGroup templates, Transition edge)
-
genSetExpr
public org.stringtemplate.v4.ST genSetExpr(org.stringtemplate.v4.STGroup templates, IntSet set, int k, boolean partOfDFA)
For intervals such as [3..3, 30..35], generate an expression that tests the lookahead similar to LA(1)==3 || (LA(1)>=30&&LA(1)<=35)
-
genTokenTypeConstants
protected void genTokenTypeConstants(org.stringtemplate.v4.ST code)
Set attributes tokens and literals attributes in the incoming code template. This is not the token vocab interchange file, but rather a list of token type ID needed by the recognizer.
-
genTokenTypeNames
protected void genTokenTypeNames(org.stringtemplate.v4.ST code)
Generate a token names table that maps token type to a printable name: either the label like INT or the literal like "begin".
-
getTokenTypeAsTargetLabel
public java.lang.String getTokenTypeAsTargetLabel(int ttype)
Get a meaningful name for a token type useful during code generation. Literals without associated names are converted to the string equivalent of their integer values. Used to generate x==ID and x==34 type comparisons etc... Essentially we are looking for the most obvious way to refer to a token type in the generated code. If in the lexer, return the char literal translated to the target language. For example, ttype=10 will yield '\n' from the getTokenDisplayName method. That must be converted to the target languages literals. For most C-derived languages no translation is needed.
-
genTokenVocabOutput
protected org.stringtemplate.v4.ST genTokenVocabOutput()
Generate a token vocab file with all the token names/types. For example: ID=7 FOR=8 'for'=8 This is independent of the target language; used by antlr internally
-
translateAction
public java.util.List<? extends java.lang.Object> translateAction(java.lang.String ruleName, GrammarAST actionTree)
-
translateArgAction
public java.util.List<org.stringtemplate.v4.ST> translateArgAction(java.lang.String ruleName, GrammarAST actionTree)
Translate an action like [3,"foo",a[3]] and return a List of the translated actions. Because actions are themselves translated to a list of chunks, must cat together into a ST>. Don't translate to strings early as we need to eval templates in context.
-
getListOfArgumentsFromAction
public static java.util.List<java.lang.String> getListOfArgumentsFromAction(java.lang.String actionText, int separatorChar)
-
getListOfArgumentsFromAction
public static int getListOfArgumentsFromAction(java.lang.String actionText, int start, int targetChar, int separatorChar, java.util.List<java.lang.String> args)
Given an arg action like [x, (*a).foo(21,33), 3.2+1, '\n', "a,oo\nick", {bl, "fdkj"eck}, ["cat\n,", x, 43]] convert to a list of arguments. Allow nested square brackets etc... Set separatorChar to ';' or ',' or whatever you want.
-
translateTemplateConstructor
public org.stringtemplate.v4.ST translateTemplateConstructor(java.lang.String ruleName, int outerAltNum, org.antlr.runtime.Token actionToken, java.lang.String templateActionText)
Given a template constructor action like %foo(a={...}) in an action, translate it to the appropriate template constructor from the templateLib. This translates a *piece* of the action.
-
issueInvalidScopeError
public void issueInvalidScopeError(java.lang.String x, java.lang.String y, Rule enclosingRule, org.antlr.runtime.Token actionToken, int outerAltNum)
-
issueInvalidAttributeError
public void issueInvalidAttributeError(java.lang.String x, java.lang.String y, Rule enclosingRule, org.antlr.runtime.Token actionToken, int outerAltNum)
-
issueInvalidAttributeError
public void issueInvalidAttributeError(java.lang.String x, Rule enclosingRule, org.antlr.runtime.Token actionToken, int outerAltNum)
-
getTemplates
public org.stringtemplate.v4.STGroup getTemplates()
-
getBaseTemplates
public org.stringtemplate.v4.STGroup getBaseTemplates()
-
setDebug
public void setDebug(boolean debug)
-
setTrace
public void setTrace(boolean trace)
-
setProfile
public void setProfile(boolean profile)
-
getRecognizerST
public org.stringtemplate.v4.ST getRecognizerST()
-
getRecognizerFileName
public java.lang.String getRecognizerFileName(java.lang.String name, int type)
Generate TParser.java and TLexer.java from T.g if combined, else just use T.java as output regardless of type.
-
getVocabFileName
public java.lang.String getVocabFileName()
What is the name of the vocab file generated for this grammar? Returns null if no .tokens file should be generated.
-
write
public void write(org.stringtemplate.v4.ST code, java.lang.String fileName) throws java.io.IOException
- Throws:
java.io.IOException
-
canGenerateSwitch
protected boolean canGenerateSwitch(DFAState s)
You can generate a switch rather than if-then-else for a DFA state if there are no semantic predicates and the number of edge label values is small enough; e.g., don't generate a switch for a state containing an edge label such as 20..52330 (the resulting byte codes would overflow the method 65k limit probably).
-
createUniqueLabel
public java.lang.String createUniqueLabel(java.lang.String name)
Create a label to track a token / rule reference's result. Technically, this is a place where I break model-view separation as I am creating a variable name that could be invalid in a target language, however, label ::= <ID><INT> is probably ok in all languages we care about.
-
-