eu.project.ttc.tools

Class TermSuitePipeline

java.lang.Object

eu.project.ttc.tools.TermSuitePipeline

public class TermSuitePipeline
extends java.lang.Object

A collection reader and ae aggregator (builder pattern) that creates and runs a full pipeline.

Method Summary

Methods

Modifier and Type	Method and Description
TermSuitePipeline	aeChineseTokenizer() Tokenizer for chinese collections.
TermSuitePipeline	aeCompostSplitter()
TermSuitePipeline	aeCompoundSplitter() Deprecated. Use aeCompostSplitter() instead
TermSuitePipeline	aeContextualizer(int scope, boolean allTerms) Computes the Contextualizer vector of all single-word terms in the term index.
TermSuitePipeline	aeExtensionDetector() Detects all inclusion/extension relation between terms that have size >= 2.
TermSuitePipeline	aeGraphicalVariantGatherer()
TermSuitePipeline	aeMateTaggerLemmatizer()
TermSuitePipeline	aeMaxSizeThresholdCleaner(TermProperty property, int maxSize)
TermSuitePipeline	aeNeoClassicalSplitter() Deprecated. Use aeCompostSplitter() instead
TermSuitePipeline	aePrefixSplitter() Deprecated. Use aeCompostSplitter() instead
TermSuitePipeline	aePrimaryOccurrenceDetector(int detectionStrategy)
TermSuitePipeline	aeRegexSpotter() The single-word and multi-word term spotter AE base on UIMA Tokens Regex.
TermSuitePipeline	aeSpecificityComputer() Computes TermProperty.WR values (and additional term properties of type TermProperty in the future).
TermSuitePipeline	aeStemmer()
TermSuitePipeline	aeStopWordsFilter() Removes from the term index any term having a stop word at its boundaries.
TermSuitePipeline	aeSyntacticVariantGatherer() Gathers terms according to their syntactic structures.
TermSuitePipeline	aeTermClassifier(TermProperty sortingProperty)
TermSuitePipeline	aeThresholdCleaner(TermProperty property, float threshold)
TermSuitePipeline	aeThresholdCleaner(TermProperty property, float threshold, boolean isPeriodic, int cleaningPeriod, int termIndexSizeTrigger)
TermSuitePipeline	aeThresholdCleanerPeriodic(TermProperty property, float threshold, int cleaningPeriod)
TermSuitePipeline	aeThresholdCleanerSizeTrigger(TermProperty property, float threshold, int termIndexSizeTrigger)
TermSuitePipeline	aeTopNCleaner(TermProperty property, int n)
TermSuitePipeline	aeTopNCleanerPeriodic(TermProperty property, int n, boolean isPeriodic, int cleaningPeriod)
TermSuitePipeline	aeTreeTagger()
TermSuitePipeline	aeUrlFilter() Filters out URLs from CAS.
TermSuitePipeline	aeWordTokenizer()
static TermSuitePipeline	create(java.lang.String lang)
static TermSuitePipeline	create(TermIndex termIndex)
org.apache.uima.analysis_engine.AnalysisEngineDescription	createDescription()
TermSuitePipeline	emptyCollection()
TermSuitePipeline	emptyTermIndex(java.lang.String name) Creates a new in-memory TermIndex on which this piepline with run.
TermSuitePipeline	enableSyntacticLabels()
TermIndex	getTermIndex() Returns the term index produced (or last modified) by this pipeline.
TermSuitePipeline	haeCasStatCounter(java.lang.String statName)
TermSuitePipeline	haeCompoundExporter(java.lang.String toFilePath) Exports all compound words of the terminology to given file path.
TermSuitePipeline	haeEval(java.lang.String refFileURI, java.lang.String outputFile, java.lang.String customLogHeader, java.lang.String rFile, java.lang.String evalTraceName, boolean rtlWithVariants)
TermSuitePipeline	haeEvalExporter(java.lang.String toFilePath, boolean withVariants)
TermSuitePipeline	haeExportVariationRuleExamples(java.lang.String toFilePath) Exports examples of matching pairs for each variation rule.
TermSuitePipeline	haeJsonExporter(java.lang.String toFilePath)
TermSuitePipeline	haeLogOverlappingRules()
TermSuitePipeline	haeScorifier() Compute variant scores recursively.
TermSuitePipeline	haeSpotterTSVWriter(java.lang.String toDirectoryPath) Export all CAS in TSV format to a given directory.
TermSuitePipeline	haeTbxExporter(java.lang.String toFilePath)
TermSuitePipeline	haeTraceTimePerf(java.lang.String toFile) Exports time progress to TSV file.
TermSuitePipeline	haeTsvExporter(java.lang.String toFilePath) Exports the TermIndex in tsv format
TermSuitePipeline	haeVariantEvalExporter(java.lang.String toFilePath, int topN, int maxVariantsPerTerm) Creates a tsv output with : - the occurrence list of each term and theirs in-text contexts
TermSuitePipeline	haeXmiCasExporter(java.lang.String toDirectoryPath) Exports all CAS as XMI files to a given directory.
org.apache.uima.resource.ExternalResourceDescription	resScoredModel()
org.apache.uima.resource.ExternalResourceDescription	resTermIndex()
TermSuitePipeline	run() Runs the pipeline with SimplePipeline on the CollectionReader that must have been defined.
TermSuitePipeline	run(org.apache.uima.jcas.JCas cas) Runs the pipeline with SimplePipeline without requiring a CollectionReader to be defined.
TermSuitePipeline	setCollection(TermSuiteCollection termSuiteCollection, java.lang.String collectionPath, java.lang.String collectionEncoding) Creates a collection reader for this pipeline.
TermSuitePipeline	setCollection(TermSuiteCollection termSuiteCollection, java.lang.String collectionPath, java.lang.String collectionEncoding, java.lang.String droppedTags, java.lang.String txtTags) Creates a collection reader of type GenericXMLToTxtCollectionReader for this pipeline.
TermSuitePipeline	setCompostCoeffs(float alpha, float beta, float gamma, float delta)
TermSuitePipeline	setCompostMaxComponentNum(int compostMaxComponentNum)
TermSuitePipeline	setCompostMinComponentSize(int compostMinComponentSize)
TermSuitePipeline	setCompostScoreThreshold(float compostScoreThreshold)
TermSuitePipeline	setCompostSegmentSimilarityThreshold(java.lang.Object compostSegmentSimilarityThreshold)
TermSuitePipeline	setContextAssocRateMeasure(java.lang.String contextAssocRateMeasure)
TermSuitePipeline	setContextualizeCoTermsType(OccurrenceType contextualizeCoTermsType)
TermSuitePipeline	setContextualizeWithCoOccurrenceFrequencyThreshhold(int contextualizeWithCoOccurrenceFrequencyThreshhold)
TermSuitePipeline	setContextualizeWithTermClasses(boolean contextualizeWithTermClasses)
TermSuitePipeline	setExportFilteringRule(java.lang.String exportFilteringRule)
TermSuitePipeline	setExportFilteringThreshold(float exportFilteringThreshold)
TermSuitePipeline	setExportJsonWithContext(boolean b)
TermSuitePipeline	setExportJsonWithOccurrences(boolean exportJsonWithOccurrences)
TermSuitePipeline	setGraphicalVariantSimilarityThreshold(float th)
TermSuitePipeline	setInlineString(java.lang.String text)
TermSuitePipeline	setKeepVariantsWhileCleaning(boolean keepVariantsWhileCleaning)
TermSuitePipeline	setMateModelPath(java.lang.String path)
TermSuitePipeline	setPostProcessingStrategy(java.lang.String postProcessingStrategy) Sets the post processing strategy for RegexSpotter analysis engine
TermSuitePipeline	setResourcePath(java.lang.String resourcePath)
TermSuitePipeline	setSpotWithOccurrences(boolean b)
TermSuitePipeline	setSyntacticRegexesFilePath(java.lang.String syntacticRegexesFilePath)
TermSuitePipeline	setTermIndex(TermIndex termIndex) Sets the term index on which this pipeline will run.
TermSuitePipeline	setTreeTaggerHome(java.lang.String treeTaggerPath)
TermSuitePipeline	setTsvExportProperties(TermProperty... properties) Defines the term properties that appear in tsv export file
TermSuitePipeline	setTsvShowHeaders(boolean tsvWithHeaders) Configures tsvExporter to (not) show headers on the first line.
TermSuitePipeline	setTsvShowScores(boolean tsvWithVariantScores) Configures tsvExporter to (not) show variant scores with the "V" label
TermSuitePipeline	setYamlVariantRulesFilePath(java.lang.String yamlVariantRulesFilePath)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

create

public static TermSuitePipeline create(java.lang.String lang)

create

public static TermSuitePipeline create(TermIndex termIndex)

run

public TermSuitePipeline run()

Runs the pipeline with SimplePipeline on the CollectionReader that must have been defined.

Throws:

TermSuitePipelineException - if no CollectionReader has been declared on this pipeline

run

public TermSuitePipeline run(org.apache.uima.jcas.JCas cas)

Runs the pipeline with SimplePipeline without requiring a CollectionReader to be defined.

Parameters:

cas - the JCas on which the pipeline operates.

setInlineString

public TermSuitePipeline setInlineString(java.lang.String text)

setCollection

public TermSuitePipeline setCollection(TermSuiteCollection termSuiteCollection,
                              java.lang.String collectionPath,
                              java.lang.String collectionEncoding)

Creates a collection reader for this pipeline.

Parameters:

termSuiteCollection -

collectionPath -

collectionEncoding -

Returns:

setCollection

public TermSuitePipeline setCollection(TermSuiteCollection termSuiteCollection,
                              java.lang.String collectionPath,
                              java.lang.String collectionEncoding,
                              java.lang.String droppedTags,
                              java.lang.String txtTags)

Creates a collection reader of type GenericXMLToTxtCollectionReader for this pipeline. Requires a list of dropped tags and txt tags for collection parsing.

Parameters:

termSuiteCollection -

collectionPath -

collectionEncoding -

droppedTags -

txtTags -

Returns:

See Also:

AbstractToTxtSaxHandler

setResourcePath

public TermSuitePipeline setResourcePath(java.lang.String resourcePath)

setContextAssocRateMeasure

public TermSuitePipeline setContextAssocRateMeasure(java.lang.String contextAssocRateMeasure)

emptyCollection

public TermSuitePipeline emptyCollection()

createDescription

public org.apache.uima.analysis_engine.AnalysisEngineDescription createDescription()

aeWordTokenizer

public TermSuitePipeline aeWordTokenizer()

aeTreeTagger

public TermSuitePipeline aeTreeTagger()

setMateModelPath

public TermSuitePipeline setMateModelPath(java.lang.String path)

aeMateTaggerLemmatizer

public TermSuitePipeline aeMateTaggerLemmatizer()

setTsvExportProperties

public TermSuitePipeline setTsvExportProperties(TermProperty... properties)

Defines the term properties that appear in tsv export file

Parameters:

properties -

Returns:

See Also:

haeTsvExporter(String)

haeTsvExporter

public TermSuitePipeline haeTsvExporter(java.lang.String toFilePath)

Exports the TermIndex in tsv format

Parameters:

toFilePath -

Returns:

See Also:

setTsvExportProperties(TermProperty...)

haeScorifier

public TermSuitePipeline haeScorifier()

Compute variant scores recursively.

Returns:

the pipeline

haeExportVariationRuleExamples

public TermSuitePipeline haeExportVariationRuleExamples(java.lang.String toFilePath)

Exports examples of matching pairs for each variation rule.

Parameters:

toFilePath - the file path where to write the examples for each variation rules

Returns:

the pipeline

haeCompoundExporter

public TermSuitePipeline haeCompoundExporter(java.lang.String toFilePath)

Exports all compound words of the terminology to given file path.

Parameters:

toFilePath -

Returns:

haeTbxExporter

public TermSuitePipeline haeTbxExporter(java.lang.String toFilePath)

haeEvalExporter

public TermSuitePipeline haeEvalExporter(java.lang.String toFilePath,
                                boolean withVariants)

setExportJsonWithOccurrences

public TermSuitePipeline setExportJsonWithOccurrences(boolean exportJsonWithOccurrences)

setExportJsonWithContext

public TermSuitePipeline setExportJsonWithContext(boolean b)

haeJsonExporter

public TermSuitePipeline haeJsonExporter(java.lang.String toFilePath)

haeVariantEvalExporter

public TermSuitePipeline haeVariantEvalExporter(java.lang.String toFilePath,
                                       int topN,
                                       int maxVariantsPerTerm)

Creates a tsv output with : - the occurrence list of each term and theirs in-text contexts. - a json structure for the evaluation of each variant

Parameters:

toFilePath - The output file path

topN - The number of variants to keep in the file

maxVariantsPerTerm - The maximum number of variants to eval for each term

Returns:

aeStemmer

public TermSuitePipeline aeStemmer()

aeRegexSpotter

public TermSuitePipeline aeRegexSpotter()

The single-word and multi-word term spotter AE base on UIMA Tokens Regex.

Returns:

aeCompoundSplitter

public TermSuitePipeline aeCompoundSplitter()

Deprecated. Use aeCompostSplitter() instead

Naive morphological analysis of compounds based on a compound dictionary resource

Returns:

aeNeoClassicalSplitter

public TermSuitePipeline aeNeoClassicalSplitter()

Deprecated. Use aeCompostSplitter() instead

Naive morphological analysis of neo-classical compounds based on a neo-classical dictionary resource

Returns:

aePrefixSplitter

public TermSuitePipeline aePrefixSplitter()

Deprecated. Use aeCompostSplitter() instead

Naive morphological analysis of prefix compounds based on a prefix dictionary resource

Returns:

aeStopWordsFilter

public TermSuitePipeline aeStopWordsFilter()

Removes from the term index any term having a stop word at its boundaries.

Returns:

See Also:

TermIndexBlacklistWordFilterAE

haeXmiCasExporter

public TermSuitePipeline haeXmiCasExporter(java.lang.String toDirectoryPath)

Exports all CAS as XMI files to a given directory.

Parameters:

toDirectoryPath -

Returns:

haeSpotterTSVWriter

public TermSuitePipeline haeSpotterTSVWriter(java.lang.String toDirectoryPath)

Export all CAS in TSV format to a given directory.

Parameters:

toDirectoryPath -

Returns:

See Also:

SpotterTSVWriter

aeChineseTokenizer

public TermSuitePipeline aeChineseTokenizer()

Tokenizer for chinese collections.

Returns:

See Also:

ChineseSegmenter

resScoredModel

public org.apache.uima.resource.ExternalResourceDescription resScoredModel()

resTermIndex

public org.apache.uima.resource.ExternalResourceDescription resTermIndex()

getTermIndex

public TermIndex getTermIndex()

Returns the term index produced (or last modified) by this pipeline.

Returns:

setTermIndex

public TermSuitePipeline setTermIndex(TermIndex termIndex)

Sets the term index on which this pipeline will run.

Parameters:

termIndex -

emptyTermIndex

public TermSuitePipeline emptyTermIndex(java.lang.String name)

Creates a new in-memory TermIndex on which this piepline with run.

Parameters:

name - the name of the new term index

Returns:

this TermSuitePipeline object

aeSpecificityComputer

public TermSuitePipeline aeSpecificityComputer()

Computes TermProperty.WR values (and additional term properties of type TermProperty in the future).

Returns:

See Also:

TermSpecificityComputer, TermProperty

setContextualizeCoTermsType

public TermSuitePipeline setContextualizeCoTermsType(OccurrenceType contextualizeCoTermsType)

setContextualizeWithTermClasses

public TermSuitePipeline setContextualizeWithTermClasses(boolean contextualizeWithTermClasses)

setContextualizeWithCoOccurrenceFrequencyThreshhold

public TermSuitePipeline setContextualizeWithCoOccurrenceFrequencyThreshhold(int contextualizeWithCoOccurrenceFrequencyThreshhold)

aeContextualizer

public TermSuitePipeline aeContextualizer(int scope,
                                 boolean allTerms)

Computes the Contextualizer vector of all single-word terms in the term index.

Parameters:

scope -

allTerms -

Returns:

See Also:

Contextualizer

aeMaxSizeThresholdCleaner

public TermSuitePipeline aeMaxSizeThresholdCleaner(TermProperty property,
                                          int maxSize)

aeThresholdCleaner

public TermSuitePipeline aeThresholdCleaner(TermProperty property,
                                   float threshold,
                                   boolean isPeriodic,
                                   int cleaningPeriod,
                                   int termIndexSizeTrigger)

aePrimaryOccurrenceDetector

public TermSuitePipeline aePrimaryOccurrenceDetector(int detectionStrategy)

aeThresholdCleanerPeriodic

public TermSuitePipeline aeThresholdCleanerPeriodic(TermProperty property,
                                           float threshold,
                                           int cleaningPeriod)

Parameters:

property -

threshold -

cleaningPeriod -

Returns:

aeThresholdCleanerSizeTrigger

public TermSuitePipeline aeThresholdCleanerSizeTrigger(TermProperty property,
                                              float threshold,
                                              int termIndexSizeTrigger)

setKeepVariantsWhileCleaning

public TermSuitePipeline setKeepVariantsWhileCleaning(boolean keepVariantsWhileCleaning)

aeThresholdCleaner

public TermSuitePipeline aeThresholdCleaner(TermProperty property,
                                   float threshold)

aeTopNCleaner

public TermSuitePipeline aeTopNCleaner(TermProperty property,
                              int n)

aeTopNCleanerPeriodic

public TermSuitePipeline aeTopNCleanerPeriodic(TermProperty property,
                                      int n,
                                      boolean isPeriodic,
                                      int cleaningPeriod)

Parameters:

property -

n -

isPeriodic -

cleaningPeriod -

Returns:

setGraphicalVariantSimilarityThreshold

public TermSuitePipeline setGraphicalVariantSimilarityThreshold(float th)

aeGraphicalVariantGatherer

public TermSuitePipeline aeGraphicalVariantGatherer()

aeUrlFilter

public TermSuitePipeline aeUrlFilter()

Filters out URLs from CAS.

Returns:

aeSyntacticVariantGatherer

public TermSuitePipeline aeSyntacticVariantGatherer()

Gathers terms according to their syntactic structures.

Returns:

aeExtensionDetector

public TermSuitePipeline aeExtensionDetector()

Detects all inclusion/extension relation between terms that have size >= 2.

Returns:

setExportFilteringRule

public TermSuitePipeline setExportFilteringRule(java.lang.String exportFilteringRule)

setExportFilteringThreshold

public TermSuitePipeline setExportFilteringThreshold(float exportFilteringThreshold)

setTreeTaggerHome

public TermSuitePipeline setTreeTaggerHome(java.lang.String treeTaggerPath)

setSyntacticRegexesFilePath

public TermSuitePipeline setSyntacticRegexesFilePath(java.lang.String syntacticRegexesFilePath)

haeLogOverlappingRules

public TermSuitePipeline haeLogOverlappingRules()

enableSyntacticLabels

public TermSuitePipeline enableSyntacticLabels()

setYamlVariantRulesFilePath

public TermSuitePipeline setYamlVariantRulesFilePath(java.lang.String yamlVariantRulesFilePath)

setCompostCoeffs

public TermSuitePipeline setCompostCoeffs(float alpha,
                                 float beta,
                                 float gamma,
                                 float delta)

setCompostMaxComponentNum

public TermSuitePipeline setCompostMaxComponentNum(int compostMaxComponentNum)

setCompostMinComponentSize

public TermSuitePipeline setCompostMinComponentSize(int compostMinComponentSize)

setCompostScoreThreshold

public TermSuitePipeline setCompostScoreThreshold(float compostScoreThreshold)

setCompostSegmentSimilarityThreshold

public TermSuitePipeline setCompostSegmentSimilarityThreshold(java.lang.Object compostSegmentSimilarityThreshold)

aeCompostSplitter

public TermSuitePipeline aeCompostSplitter()

haeCasStatCounter

public TermSuitePipeline haeCasStatCounter(java.lang.String statName)

haeTraceTimePerf

public TermSuitePipeline haeTraceTimePerf(java.lang.String toFile)

Exports time progress to TSV file. Columns are :

• elapsed time from initialization in milliseconds
• number of docs processed
• cumulated size of data processed
• number of terms in term index
• number of WordAnnotation processed

Parameters:

toFile -

Returns:

aeTermClassifier

public TermSuitePipeline aeTermClassifier(TermProperty sortingProperty)

Parameters:

sortingProperty - the term property used to order terms before they are classified. The first term of a class appearing given this order will be considered as the head of the class.

Returns:

See Also:

TermClassifier

haeEval

public TermSuitePipeline haeEval(java.lang.String refFileURI,
                        java.lang.String outputFile,
                        java.lang.String customLogHeader,
                        java.lang.String rFile,
                        java.lang.String evalTraceName,
                        boolean rtlWithVariants)

Parameters:

refFileURI - The path to reference termino

outputFile - The path to output log file

customLogHeader - A custom string to add in the header of the output log file

rFile - The path to output r file

evalTraceName - The name of the eval trace

rtlWithVariants - true if variants of the reference termino should be kept during the eval

Returns:

setSpotWithOccurrences

public TermSuitePipeline setSpotWithOccurrences(boolean b)

setPostProcessingStrategy

public TermSuitePipeline setPostProcessingStrategy(java.lang.String postProcessingStrategy)

Sets the post processing strategy for RegexSpotter analysis engine

Parameters:

postProcessingStrategy -

Returns:

See Also:

aeRegexSpotter(), OccurrenceBuffer.NO_CLEANING, OccurrenceBuffer.KEEP_PREFIXES, OccurrenceBuffer.KEEP_SUFFIXES

setTsvShowHeaders

public TermSuitePipeline setTsvShowHeaders(boolean tsvWithHeaders)

Configures tsvExporter to (not) show headers on the first line.

Parameters:

tsvWithHeaders - the flag

Returns:

this pipeline builder

setTsvShowScores

public TermSuitePipeline setTsvShowScores(boolean tsvWithVariantScores)

Configures tsvExporter to (not) show variant scores with the "V" label

Parameters:

tsvWithVariantScores - the flag

Returns:

this pipeline builder