org.allenai.nlpstack.parse.poly
reranking
Applies a ParseNeighborhoodExtractor to all parses in a PolytreeParseSource and returns an iterator over all of the extracted neighborhoods.
Creates a feature for every keyword (from a dictionary of keywords) that appears in the input neighborhood.
Rescores a parse tree based on a linear combination of features.
Rescores a parse tree based on a linear combination of features.
computes a feature vector from the parse tree
computes a linear combination of the computed features
Collects statistics over "neighborhood events."
Collects statistics over "neighborhood events."
An example might help. A neighborhood is a collection of tokens, e.g. a node and its children in a dependency parse. A neighborhood event is a mapping of these tokens to a sequence of strings, e.g. we might map each token to its part-of-speech tag.
Given a corpus of dependency parses, we might want to collect a histogram that tells us how many times each neighborhood event like (VERB, NOUN, NOUN) occurs in the corpus. This is what the NeighborhoodEventStatistic does.
provides a stream of neighborhoods
a transformation from neighborhoods to events
Maps a parse tree node to one or more of its neighborhoods.
Maps a parse tree node to one or more of its neighborhoods.
Different extractors will define "neighborhood" in different ways. For instance, one might want to consider neighborhoods like: - a node and its children - a node and its parents - a node and its breadcrumb
A NeighborhoodTransform maps a Neighborhood to zero or more feature names.
A NeighborhoodTransform maps a Neighborhood to zero or more feature names.
An example might help. Suppose that we have a neighborhood consisting of (node, child1, child2), i.e. three nodes of a parse tree. A transform might map these to the sequence of their POS tags, e.g. FeatureName(Seq('VERB, 'NOUN, 'NOUN)).
Applies a NeighborhoodExtractor to all tokens in a parse tree and returns an iterator over all of the extracted neighborhoods.
Maps a selected node of a parse tree into a feature vector.
A ParseNodeFeatureUnion merges the output of a list of features.
A ParseNodeFeatureUnion merges the output of a list of features.
a list of the features we want to merge into a single feature
Rescores a parse tree according to a specified scoring function.
Rescores a parse tree according to a specified scoring function.
the desired scoring function
Maps a scored parse into a feature vector.
A PolytreeParseFeatureUnion merges the output of a list of features.
A PolytreeParseFeatureUnion merges the output of a list of features.
a list of the features we want to merge into a single feature
Maps the tokens of a neighborhood to a particular property in their token's property map.
Maps the tokens of a neighborhood to a particular property in their token's property map.
name of the desired property
Extracts neighborhood (child_k, token), where child_k is the kth child of the input token.
Extracts neighborhood (child_k, token), where child_k is the kth child of the input token.
If the input token does not have a kth child, the apply operation will return the empty sequence.
Extracts neighborhood (parent_k, token), where parent_k is the kth parent of the input token.
Extracts neighborhood (parent_k, token), where parent_k is the kth parent of the input token.
If the input token does not have a kth parent, the apply operation will return the empty sequence.
Extracts neighborhood (child_k), where child_k is the kth child of the input token.
Extracts neighborhood (child_k), where child_k is the kth child of the input token.
If the input token does not have a kth child, the apply operation will return the empty sequence.
Extracts neighborhood (parent_k), where parent_k is the kth parent of the input token.
Extracts neighborhood (parent_k), where parent_k is the kth parent of the input token.
If the input token does not have a kth parent, the apply operation will return the empty sequence.
Creates a feature for every suffix (from a dictionary of suffixes) that appears in the input neighborhood.
Creates a feature for every suffix (from a dictionary of suffixes) that appears in the input neighborhood.
the set of suffixes to consider
A TransformedNeighborhoodFeature creates a feature vector from a set of neighborhood extractors and transforms.
A TransformedNeighborhoodFeature creates a feature vector from a set of neighborhood extractors and transforms.
the neighborhood extractors you want to apply to each parse node
the transforms you want to apply to the extracted neighborhoods
Creates a feature per frame for all verbnet frames corresponding to the tokens in the input neighborhood.
Creates a feature per frame for all verbnet frames corresponding to the tokens in the input neighborhood.
the Verbnet utility object to query
This reranking function attempts to rerank parses based on how many "weird" nodes they have, according to a "weirdness" classifier.
This reranking function attempts to rerank parses based on how many "weird" nodes they have, according to a "weirdness" classifier.
the weirdness classifier
computes a feature vector from a parse tree node
the minimum probability of weirdness in order for a node to count as "weird"
A parse statistic that collects weirdness statistics for candidate parses.
A parse statistic that collects weirdness statistics for candidate parses.
the weirdness reranking function
Extracts the neighborhood (child1, ..., childK) from a parse tree, where childI is the Ith child of the input token.
Extracts the neighborhood (parent1, ..., parentK) from a parse tree, where parentI is the Ith parent of the input token.
Creates a feature for the label on the arc connecting two tokens in a two-token neighborhood.
Creates a feature for the label on the arc connecting two tokens in a two-token neighborhood.
Note that the apply operator will throw an exception if the argument neighborhood does not have exactly two tokens. It will also throw an exception if the parse does not contains an arc between the two neighborhood nodes.
Simply passes along the original score of the parse as a feature.
Creates a feature describing the cardinality of a neighborhood, i.e.
Creates a feature describing the cardinality of a neighborhood, i.e. the number of tokens in the neighborhood.
Creates a feature describing the order of two tokens in a two-token neighborhood.
Creates a feature describing the order of two tokens in a two-token neighborhood.
Specifically, it will be 'L if the first token appears to the left of the second token in the sentence. Otherwise, it will be 'R.
Note that the apply operator will throw an exception if the argument neighborhood does not have exactly two tokens. It will also throw an exception if the two tokens are the same (i.e. neither appears to the left of the other).
Extracts all neighborhoods of the form (child) from a parse tree, where child is one of the children of the input token.
Extracts all neighborhoods of the form (parent) from a parse tree, where parent is one of the parents of the input token.
This command-line takes a serialized reranking function and uses it to rerank an n-best list.
This command-line takes a serialized reranking function and uses it to rerank an n-best list. The resulting parses are then evaluated against a gold set.
A command-line that shows the way towards possibly better parse reranking.
A command-line that shows the way towards possibly better parse reranking.
This trains a "weirdness" classifier that learns to classify parse tree nodes as "weird" or not "weird," which can be used to rerank parses based on how many of their nodes are classified as "weird."
A command-line that plots an accuracy/yield curve, given a reranking function and a parser.
Extracts neighborhood (token) from the parse tree.
Simply passes along the length of the sentence as a feature.
Creates a feature for every keyword (from a dictionary of keywords) that appears in the input neighborhood.
Note that the keyword matching is case-insensitive.
the set of words to consider