org.python.indexer

Class Indexer

java.lang.Object

org.python.indexer.Indexer

public class Indexer
extends java.lang.Object

Indexes a set of Python files and builds a code graph.

This class is not thread-safe.

Field Summary

Fields

Modifier and Type	Field and Description
Builtins	builtins Manages the built-in modules -- that is, modules from the standard Python library that are implemented in C and consequently have no Python source.
java.lang.String	currentFile
java.util.Set<java.lang.String>	failedModules When resolving imports we look in various possible locations.
Scope	globaltable The top-level (builtin) scope.
static Indexer	idx The global indexer instance.
Scope	moduleTable A scope containing bindings for all modules currently loaded by the indexer.
java.util.Map<java.lang.String,java.util.List<Diagnostic>>	parseErrs
java.util.List<java.lang.String>	path
java.util.Map<java.lang.String,java.util.List<Diagnostic>>	problems Diagnostics.
java.lang.String	projDir

Constructor Summary

Constructors

Constructor and Description
Indexer()

Method Summary

Modifier and Type	Method and Description
void	addPath(java.lang.String p) Adds the specified absolute path to the module search path.
void	addPaths(java.util.List<java.lang.String> p) Adds the specified absolute paths to the module search path.
boolean	aggressiveAssertionsEnabled()
void	clearAstCache() Clears the AST cache (to free up memory).
void	clearModuleTable() Clears the module table, discarding all resolved ASTs (modules) and their scope information.
void	enableAggressiveAssertions(boolean enable) Configures whether the indexer should abort with an exception when it encounters an internal error or unexpected program state.
void	fine(java.lang.String msg)
void	finer(java.lang.String msg)
java.util.List<Outliner.Entry>	generateOutline(java.lang.String path) Create an outline for a file in the index.
NModule	getAstForFile(java.lang.String file) Returns the syntax tree for file.
NModule	getAstForFile(java.lang.String file, java.lang.String contents) Returns the syntax tree for file.
java.util.Map<java.lang.String,NBinding>	getBindings() Returns the mutable set of all bindings collected, keyed on their qnames.
NModuleType	getBuiltinModule(java.lang.String qname)
java.util.List<Diagnostic>	getDiagnosticsForFile(java.lang.String file) Returns the list, possibly empty but never null, of errors and warnings generated in the file.
java.util.List<java.lang.String>	getLoadedFiles()
java.util.List<java.lang.String>	getLoadPath() Returns the module search path -- the project directory followed by any paths that were added by addPath(String).
java.util.logging.Logger	getLogger()
NModuleType	getModuleForFile(java.lang.String file) Returns (loading/resolving if necessary) the module for a given source path.
java.lang.String	getStatusReport() Report resolution rate and other statistics data.
void	handleException(java.lang.String msg, java.lang.Throwable cause) If aggressive assertions are enabled, propagates the passed Throwable, wrapped in an IndexingException.
void	info(java.lang.String msg)
boolean	isLibFile(java.lang.String file)
NModuleType	loadFile(java.lang.String path) Loads a file and all its ancestor packages.
NModuleType	loadFile(java.lang.String path, boolean skipChain) Load, parse and analyze a source file given its absolute path.
void	loadFileRecursive(java.lang.String fullname) Load all Python source files recursively if the given fullname is a directory; otherwise just load a file.
NModuleType	loadModule(java.lang.String modname) This method searches the module path for the module modname.
NModuleType	loadString(java.lang.String path, java.lang.String contents) Loads a module from a string containing the module contents.
void	log(java.util.logging.Level level, java.lang.String msg)
NBinding	lookupQname(java.lang.String qname) Return the binding for qname, or null if not known.
NType	lookupQnameType(java.lang.String qname) Return the type for qname, or null if not known.
int	numFilesLoaded()
NBinding	putBinding(NBinding b)
void	putLocation(NNode node, NBinding b) Add a reference to binding b at AST node node.
void	putLocation(Ref ref, NBinding b)
void	putProblem(NNode loc, java.lang.String msg)
void	putProblem(java.lang.String file, int beg, int end, java.lang.String msg)
void	ready() Performs final indexing-building passes, including marking references to undeclared variables.
void	recordUnresolvedModule(java.lang.String qname, java.lang.String file) Reports a failed module or submodule resolution.
void	release() Releases all resources for the current indexer.
void	removeBinding(NBinding b)
void	reportFailedAssertion(java.lang.String msg) Signals a failed assertion about the state of the indexer or index.
void	setLogger(java.util.logging.Logger logger)
void	setPath(java.util.List<java.lang.String> path) Sets the module search path to the specified list of absolute paths.
void	setProjectDir(java.lang.String cd)
void	severe(java.lang.String msg)
java.lang.String	toString()
void	updateLocation(Ref node, NBinding b) Add node as a reference to binding b, removing node from any other binding ref-lists that it may have occupied.
void	warn(java.lang.String msg)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

idx

public static Indexer idx

The global indexer instance. Provides convenient access to global resources, as well as easy cleanup of resources after the index is built.

moduleTable

public Scope moduleTable

A scope containing bindings for all modules currently loaded by the indexer.

globaltable

public Scope globaltable

The top-level (builtin) scope.

problems

public java.util.Map<java.lang.String,java.util.List<Diagnostic>> problems

Diagnostics.

parseErrs

public java.util.Map<java.lang.String,java.util.List<Diagnostic>> parseErrs

currentFile

public java.lang.String currentFile

projDir

public java.lang.String projDir

path

public java.util.List<java.lang.String> path

failedModules

public java.util.Set<java.lang.String> failedModules

When resolving imports we look in various possible locations. This set keeps track of modules we attempted but didn't find.

builtins

public Builtins builtins

Manages the built-in modules -- that is, modules from the standard Python library that are implemented in C and consequently have no Python source.

Constructor Detail

Indexer

public Indexer()

Method Detail

setLogger

public void setLogger(java.util.logging.Logger logger)

getLogger

public java.util.logging.Logger getLogger()

setProjectDir

public void setProjectDir(java.lang.String cd)
                   throws java.io.IOException

Throws:

java.io.IOException

enableAggressiveAssertions

public void enableAggressiveAssertions(boolean enable)

Configures whether the indexer should abort with an exception when it encounters an internal error or unexpected program state. Normally the indexer attempts to continue indexing, on the assumption that having an index with mostly good data is better than having no index at all. Enabling aggressive assertions is useful for debugging the indexer.

aggressiveAssertionsEnabled

public boolean aggressiveAssertionsEnabled()

handleException

public void handleException(java.lang.String msg,
                            java.lang.Throwable cause)

If aggressive assertions are enabled, propagates the passed Throwable, wrapped in an IndexingException.

Parameters:

msg - descriptive message; ok to be null

Throws:

IndexingException

reportFailedAssertion

public void reportFailedAssertion(java.lang.String msg)

Signals a failed assertion about the state of the indexer or index. If aggressive assertions are enabled, throws an IndexingException. Otherwise the message is logged as a warning, and indexing continues.

Parameters:

msg - a descriptive message about the problem

Throws:

IndexingException

See Also:

enableAggressiveAssertions(boolean)

addPaths

public void addPaths(java.util.List<java.lang.String> p)
              throws java.io.IOException

Adds the specified absolute paths to the module search path.

Throws:

java.io.IOException

addPath

public void addPath(java.lang.String p)
             throws java.io.IOException

Adds the specified absolute path to the module search path.

Throws:

java.io.IOException

setPath

public void setPath(java.util.List<java.lang.String> path)
             throws java.io.IOException

Sets the module search path to the specified list of absolute paths.

Throws:

java.io.IOException

getLoadPath

public java.util.List<java.lang.String> getLoadPath()

Returns the module search path -- the project directory followed by any paths that were added by addPath(String).

isLibFile

public boolean isLibFile(java.lang.String file)

getBindings

public java.util.Map<java.lang.String,NBinding> getBindings()

Returns the mutable set of all bindings collected, keyed on their qnames.

lookupQname

public NBinding lookupQname(java.lang.String qname)

Return the binding for qname, or null if not known.

lookupQnameType

public NType lookupQnameType(java.lang.String qname)

Return the type for qname, or null if not known.

Throws:

java.lang.IllegalStateException - if ready() has not been called.

getModuleForFile

public NModuleType getModuleForFile(java.lang.String file)
                             throws java.lang.Exception

Returns (loading/resolving if necessary) the module for a given source path.

Parameters:

file - absolute file path

Throws:

java.lang.Exception

getDiagnosticsForFile

public java.util.List<Diagnostic> getDiagnosticsForFile(java.lang.String file)

Returns the list, possibly empty but never null, of errors and warnings generated in the file.

generateOutline

public java.util.List<Outliner.Entry> generateOutline(java.lang.String path)
                                               throws java.lang.Exception

Create an outline for a file in the index.

Parameters:

path - the file for which to build the outline

Returns:

a list of entries constituting the file outline. Returns an empty list if the indexer hasn't indexed that path.

Throws:

java.lang.Exception

putLocation

public void putLocation(NNode node,
                        NBinding b)

Add a reference to binding b at AST node node.

Parameters:

node - a node referring to a name binding. Typically a NName, NStr or NUrl.

putLocation

public void putLocation(Ref ref,
                        NBinding b)

updateLocation

public void updateLocation(Ref node,
                           NBinding b)

Add node as a reference to binding b, removing node from any other binding ref-lists that it may have occupied. Currently only used in retargeting attribute references from provisional bindings once the actual binding is determined.

removeBinding

public void removeBinding(NBinding b)

putBinding

public NBinding putBinding(NBinding b)

putProblem

public void putProblem(NNode loc,
                       java.lang.String msg)

putProblem

public void putProblem(java.lang.String file,
                       int beg,
                       int end,
                       java.lang.String msg)

loadFile

public NModuleType loadFile(java.lang.String path)
                     throws java.lang.Exception

Loads a file and all its ancestor packages.

Throws:

java.lang.Exception

See Also:

loadFile(String,boolean)

loadString

public NModuleType loadString(java.lang.String path,
                              java.lang.String contents)
                       throws java.lang.Exception

Loads a module from a string containing the module contents. Idempotent: looks in the module cache first. Used for simple unit tests.

Parameters:

path - a path for reporting/caching purposes. The filename component is used to construct the module qualified name.

Throws:

java.lang.Exception

loadFile

public NModuleType loadFile(java.lang.String path,
                            boolean skipChain)
                     throws java.lang.Exception

Load, parse and analyze a source file given its absolute path. By default, loads the entire ancestor package chain.

Parameters:

path - the absolute path to the file or directory. If it is a directory, it is suffixed with "__init__.py", and only that file is loaded from the directory.

skipChain - true to skip loading ancestor packages

Returns:

null if path could not be loaded

Throws:

java.lang.Exception

getAstForFile

public NModule getAstForFile(java.lang.String file)
                      throws java.lang.Exception

Returns the syntax tree for file.

Throws:

java.lang.Exception

getAstForFile

public NModule getAstForFile(java.lang.String file,
                             java.lang.String contents)
                      throws java.lang.Exception

Returns the syntax tree for file.

Throws:

java.lang.Exception

getBuiltinModule

public NModuleType getBuiltinModule(java.lang.String qname)
                             throws java.lang.Exception

Throws:

java.lang.Exception

loadModule

public NModuleType loadModule(java.lang.String modname)
                       throws java.lang.Exception

This method searches the module path for the module modname. If found, it is passed to loadFile(java.lang.String).

The mechanisms for importing modules are in general statically undecidable. We make a reasonable effort to follow the most common lookup rules.

Parameters:

modname - a module name. Can be a relative path to a directory or a file (without the extension) or a possibly-qualified module name. If it is a module name, cannot contain leading dots.

Throws:

java.lang.Exception

See Also:

The Import Statement

loadFileRecursive

public void loadFileRecursive(java.lang.String fullname)
                       throws java.lang.Exception

Load all Python source files recursively if the given fullname is a directory; otherwise just load a file. Looks at file extension to determine whether to load a given file.

Throws:

java.lang.Exception

ready

public void ready()

Performs final indexing-building passes, including marking references to undeclared variables. Caller should invoke this method after loading all files.

clearAstCache

public void clearAstCache()

Clears the AST cache (to free up memory). Subsequent calls to getAstForFile(java.lang.String) will either fetch the serialized AST from a disk cache or re-parse the file from scratch.

clearModuleTable

public void clearModuleTable()

Clears the module table, discarding all resolved ASTs (modules) and their scope information.

recordUnresolvedModule

public void recordUnresolvedModule(java.lang.String qname,
                                   java.lang.String file)

Reports a failed module or submodule resolution.

Parameters:

qname - module qname, e.g. "org.foo.bar"

file - the file where the unresolved import occurred

getStatusReport

public java.lang.String getStatusReport()

Report resolution rate and other statistics data.

numFilesLoaded

public int numFilesLoaded()

getLoadedFiles

public java.util.List<java.lang.String> getLoadedFiles()

log

public void log(java.util.logging.Level level,
                java.lang.String msg)

severe

public void severe(java.lang.String msg)

warn

public void warn(java.lang.String msg)

info

public void info(java.lang.String msg)

fine

public void fine(java.lang.String msg)

finer

public void finer(java.lang.String msg)

release

public void release()

Releases all resources for the current indexer.

toString

public java.lang.String toString()

Overrides:

toString in class java.lang.Object