class ChunkTokenizerModel extends TokenizerModel
Instantiated model of the ChunkTokenizer. For usage and examples see the documentation of the main class.
- Grouped
- Alphabetic
- By Inheritance
- ChunkTokenizerModel
- TokenizerModel
- HasSimpleAnnotate
- AnnotatorModel
- CanBeLazy
- RawAnnotator
- HasOutputAnnotationCol
- HasInputAnnotationCols
- HasOutputAnnotatorType
- ParamsAndFeaturesWritable
- HasFeatures
- DefaultParamsWritable
- MLWritable
- Model
- Transformer
- PipelineStage
- Logging
- Params
- Serializable
- Identifiable
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
Type Members
- type AnnotationContent = Seq[Row]
internal types to show Rows as a relevant StructType Should be deleted once Spark releases UserDefinedTypes to @developerAPI
internal types to show Rows as a relevant StructType Should be deleted once Spark releases UserDefinedTypes to @developerAPI
- Attributes
- protected
- Definition Classes
- AnnotatorModel
- type AnnotatorType = String
- Definition Classes
- HasOutputAnnotatorType
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def $[T](param: Param[T]): T
- Attributes
- protected
- Definition Classes
- Params
- def $$[T](feature: StructFeature[T]): T
- Attributes
- protected
- Definition Classes
- HasFeatures
- def $$[K, V](feature: MapFeature[K, V]): Map[K, V]
- Attributes
- protected
- Definition Classes
- HasFeatures
- def $$[T](feature: SetFeature[T]): Set[T]
- Attributes
- protected
- Definition Classes
- HasFeatures
- def $$[T](feature: ArrayFeature[T]): Array[T]
- Attributes
- protected
- Definition Classes
- HasFeatures
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- def _transform(dataset: Dataset[_], recursivePipeline: Option[PipelineModel]): DataFrame
- Attributes
- protected
- Definition Classes
- AnnotatorModel
- def addSplitChars(v: String): ChunkTokenizerModel.this.type
One character string to split tokens inside, such as hyphens.
One character string to split tokens inside, such as hyphens. Ignored if using infix, prefix or suffix patterns.
- Definition Classes
- TokenizerModel
- def afterAnnotate(dataset: DataFrame): DataFrame
- Attributes
- protected
- Definition Classes
- AnnotatorModel
- def annotate(annotations: Seq[Annotation]): Seq[Annotation]
one to many annotation
one to many annotation
- annotations
Annotations that correspond to inputAnnotationCols generated by previous annotators if any
- returns
any number of annotations processed for every input annotation. Not necessary one to one relationship
- Definition Classes
- ChunkTokenizerModel → TokenizerModel → HasSimpleAnnotate
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def beforeAnnotate(dataset: Dataset[_]): Dataset[_]
- Attributes
- protected
- Definition Classes
- AnnotatorModel
- val caseSensitiveExceptions: BooleanParam
Whether to care for case sensitiveness in exceptions (Default:
true)Whether to care for case sensitiveness in exceptions (Default:
true)- Definition Classes
- TokenizerModel
- final def checkSchema(schema: StructType, inputAnnotatorType: String): Boolean
- Attributes
- protected
- Definition Classes
- HasInputAnnotationCols
- final def clear(param: Param[_]): ChunkTokenizerModel.this.type
- Definition Classes
- Params
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @HotSpotIntrinsicCandidate() @native()
- def copy(extra: ParamMap): TokenizerModel
requirement for annotators copies
requirement for annotators copies
- Definition Classes
- RawAnnotator → Model → Transformer → PipelineStage → Params
- def copyValues[T <: Params](to: T, extra: ParamMap): T
- Attributes
- protected
- Definition Classes
- Params
- final def defaultCopy[T <: Params](extra: ParamMap): T
- Attributes
- protected
- Definition Classes
- Params
- def dfAnnotate: UserDefinedFunction
Wraps annotate to happen inside SparkSQL user defined functions in order to act with org.apache.spark.sql.Column
Wraps annotate to happen inside SparkSQL user defined functions in order to act with org.apache.spark.sql.Column
- returns
udf function to be applied to inputCols using this annotator's annotate function as part of ML transformation
- Definition Classes
- HasSimpleAnnotate
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- val exceptions: StringArrayParam
Words that won't be affected by tokenization rules
Words that won't be affected by tokenization rules
- Definition Classes
- TokenizerModel
- def explainParam(param: Param[_]): String
- Definition Classes
- Params
- def explainParams(): String
- Definition Classes
- Params
- def extraValidate(structType: StructType): Boolean
- Attributes
- protected
- Definition Classes
- RawAnnotator
- def extraValidateMsg: String
Override for additional custom schema checks
Override for additional custom schema checks
- Attributes
- protected
- Definition Classes
- RawAnnotator
- final def extractParamMap(): ParamMap
- Definition Classes
- Params
- final def extractParamMap(extra: ParamMap): ParamMap
- Definition Classes
- Params
- val features: ArrayBuffer[Feature[_, _, _]]
- Definition Classes
- HasFeatures
- def get[T](feature: StructFeature[T]): Option[T]
- Attributes
- protected
- Definition Classes
- HasFeatures
- def get[K, V](feature: MapFeature[K, V]): Option[Map[K, V]]
- Attributes
- protected
- Definition Classes
- HasFeatures
- def get[T](feature: SetFeature[T]): Option[Set[T]]
- Attributes
- protected
- Definition Classes
- HasFeatures
- def get[T](feature: ArrayFeature[T]): Option[Array[T]]
- Attributes
- protected
- Definition Classes
- HasFeatures
- final def get[T](param: Param[T]): Option[T]
- Definition Classes
- Params
- def getCaseSensitiveExceptions(value: Boolean): Boolean
Whether to follow case sensitiveness for matching exceptions in text
Whether to follow case sensitiveness for matching exceptions in text
- Definition Classes
- TokenizerModel
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @HotSpotIntrinsicCandidate() @native()
- final def getDefault[T](param: Param[T]): Option[T]
- Definition Classes
- Params
- def getExceptions: Array[String]
Words that won't be affected by tokenization rules
Words that won't be affected by tokenization rules
- Definition Classes
- TokenizerModel
- def getInputCols: Array[String]
- returns
input annotations columns currently used
- Definition Classes
- HasInputAnnotationCols
- def getLazyAnnotator: Boolean
- Definition Classes
- CanBeLazy
- def getMaxLength(value: Int): Int
Set the maximum allowed length for each token
Set the maximum allowed length for each token
- Definition Classes
- TokenizerModel
- def getMinLength(value: Int): Int
Set the minimum allowed length for each token
Set the minimum allowed length for each token
- Definition Classes
- TokenizerModel
- final def getOrDefault[T](param: Param[T]): T
- Definition Classes
- Params
- final def getOutputCol: String
Gets annotation column name going to generate
Gets annotation column name going to generate
- Definition Classes
- HasOutputAnnotationCol
- def getParam(paramName: String): Param[Any]
- Definition Classes
- Params
- def getSplitChars: Array[String]
List of 1 character string to split tokens inside, such as hyphens.
List of 1 character string to split tokens inside, such as hyphens. Ignored if using infix, prefix or suffix patterns
- Definition Classes
- TokenizerModel
- def getSplitPattern: String
List of 1 character string to split tokens inside, such as hyphens.
List of 1 character string to split tokens inside, such as hyphens. Ignored if using infix, prefix or suffix patterns.
- Definition Classes
- TokenizerModel
- def getTargetPattern: String
pattern to grab from text as token candidates.
pattern to grab from text as token candidates. Defaults \\S+
- Definition Classes
- TokenizerModel
- final def hasDefault[T](param: Param[T]): Boolean
- Definition Classes
- Params
- def hasParam(paramName: String): Boolean
- Definition Classes
- Params
- def hasParent: Boolean
- Definition Classes
- Model
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @HotSpotIntrinsicCandidate() @native()
- def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
- Attributes
- protected
- Definition Classes
- Logging
- def initializeLogIfNecessary(isInterpreter: Boolean): Unit
- Attributes
- protected
- Definition Classes
- Logging
- val inputAnnotatorTypes: Array[AnnotatorType]
Output Annotator Type : CHUNK
Output Annotator Type : CHUNK
- Definition Classes
- ChunkTokenizerModel → TokenizerModel → HasInputAnnotationCols
- final val inputCols: StringArrayParam
columns that contain annotations necessary to run this annotator AnnotatorType is used both as input and output columns if not specified
columns that contain annotations necessary to run this annotator AnnotatorType is used both as input and output columns if not specified
- Attributes
- protected
- Definition Classes
- HasInputAnnotationCols
- final def isDefined(param: Param[_]): Boolean
- Definition Classes
- Params
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def isSet(param: Param[_]): Boolean
- Definition Classes
- Params
- def isTraceEnabled(): Boolean
- Attributes
- protected
- Definition Classes
- Logging
- val lazyAnnotator: BooleanParam
- Definition Classes
- CanBeLazy
- def log: Logger
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logName: String
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- val maxLength: IntParam
Set the maximum allowed length for each token
Set the maximum allowed length for each token
- Definition Classes
- TokenizerModel
- val minLength: IntParam
Set the minimum allowed length for each token
Set the minimum allowed length for each token
- Definition Classes
- TokenizerModel
- def msgHelper(schema: StructType): String
- Attributes
- protected
- Definition Classes
- HasInputAnnotationCols
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @HotSpotIntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @HotSpotIntrinsicCandidate() @native()
- def onWrite(path: String, spark: SparkSession): Unit
- Attributes
- protected
- Definition Classes
- ParamsAndFeaturesWritable
- val optionalInputAnnotatorTypes: Array[String]
- Definition Classes
- HasInputAnnotationCols
- val outputAnnotatorType: AnnotatorType
Output Annotator Type : TOKEN
Output Annotator Type : TOKEN
- Definition Classes
- ChunkTokenizerModel → TokenizerModel → HasOutputAnnotatorType
- final val outputCol: Param[String]
- Attributes
- protected
- Definition Classes
- HasOutputAnnotationCol
- lazy val params: Array[Param[_]]
- Definition Classes
- Params
- var parent: Estimator[TokenizerModel]
- Definition Classes
- Model
- val rules: StructFeature[RuleFactory]
rules
rules
- Definition Classes
- TokenizerModel
- def save(path: String): Unit
- Definition Classes
- MLWritable
- Annotations
- @throws("If the input path already exists but overwrite is not enabled.") @Since("1.6.0")
- def set[T](feature: StructFeature[T], value: T): ChunkTokenizerModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
- def set[K, V](feature: MapFeature[K, V], value: Map[K, V]): ChunkTokenizerModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
- def set[T](feature: SetFeature[T], value: Set[T]): ChunkTokenizerModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
- def set[T](feature: ArrayFeature[T], value: Array[T]): ChunkTokenizerModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
- final def set(paramPair: ParamPair[_]): ChunkTokenizerModel.this.type
- Attributes
- protected
- Definition Classes
- Params
- final def set(param: String, value: Any): ChunkTokenizerModel.this.type
- Attributes
- protected
- Definition Classes
- Params
- final def set[T](param: Param[T], value: T): ChunkTokenizerModel.this.type
- Definition Classes
- Params
- def setCaseSensitiveExceptions(value: Boolean): ChunkTokenizerModel.this.type
Whether to follow case sensitiveness for matching exceptions in text
Whether to follow case sensitiveness for matching exceptions in text
- Definition Classes
- TokenizerModel
- def setDefault[T](feature: StructFeature[T], value: () => T): ChunkTokenizerModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
- def setDefault[K, V](feature: MapFeature[K, V], value: () => Map[K, V]): ChunkTokenizerModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
- def setDefault[T](feature: SetFeature[T], value: () => Set[T]): ChunkTokenizerModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
- def setDefault[T](feature: ArrayFeature[T], value: () => Array[T]): ChunkTokenizerModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
- final def setDefault(paramPairs: ParamPair[_]*): ChunkTokenizerModel.this.type
- Attributes
- protected
- Definition Classes
- Params
- final def setDefault[T](param: Param[T], value: T): ChunkTokenizerModel.this.type
- Attributes
- protected[org.apache.spark.ml]
- Definition Classes
- Params
- def setExceptions(value: Array[String]): ChunkTokenizerModel.this.type
Words that won't be affected by tokenization rules
Words that won't be affected by tokenization rules
- Definition Classes
- TokenizerModel
- final def setInputCols(value: String*): ChunkTokenizerModel.this.type
- Definition Classes
- HasInputAnnotationCols
- def setInputCols(value: Array[String]): ChunkTokenizerModel.this.type
Overrides required annotators column if different than default
Overrides required annotators column if different than default
- Definition Classes
- HasInputAnnotationCols
- def setLazyAnnotator(value: Boolean): ChunkTokenizerModel.this.type
- Definition Classes
- CanBeLazy
- def setMaxLength(value: Int): ChunkTokenizerModel.this.type
Set the maximum allowed length for each token
Set the maximum allowed length for each token
- Definition Classes
- TokenizerModel
- def setMinLength(value: Int): ChunkTokenizerModel.this.type
Set the minimum allowed length for each token
Set the minimum allowed length for each token
- Definition Classes
- TokenizerModel
- final def setOutputCol(value: String): ChunkTokenizerModel.this.type
Overrides annotation column name when transforming
Overrides annotation column name when transforming
- Definition Classes
- HasOutputAnnotationCol
- def setParent(parent: Estimator[TokenizerModel]): TokenizerModel
- Definition Classes
- Model
- def setRules(ruleFactory: RuleFactory): ChunkTokenizerModel.this.type
Rules factory for tokenization
Rules factory for tokenization
- Definition Classes
- TokenizerModel
- def setSplitChars(v: Array[String]): ChunkTokenizerModel.this.type
List of 1 character string to split tokens inside, such as hyphens.
List of 1 character string to split tokens inside, such as hyphens. Ignored if using infix, prefix or suffix patterns.
- Definition Classes
- TokenizerModel
- def setSplitPattern(value: String): ChunkTokenizerModel.this.type
List of 1 character string to split tokens inside, such as hyphens.
List of 1 character string to split tokens inside, such as hyphens. Ignored if using infix, prefix or suffix patterns.
- Definition Classes
- TokenizerModel
- def setTargetPattern(value: String): ChunkTokenizerModel.this.type
pattern to grab from text as token candidates.
pattern to grab from text as token candidates. Defaults \\S+
- Definition Classes
- TokenizerModel
- val splitChars: StringArrayParam
character list used to separate from the inside of tokens
character list used to separate from the inside of tokens
- Definition Classes
- TokenizerModel
- val splitPattern: Param[String]
pattern to separate from the inside of tokens.
pattern to separate from the inside of tokens. takes priority over splitChars.
- Definition Classes
- TokenizerModel
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def tag(sentences: Seq[Sentence]): Seq[TokenizedSentence]
This func generates a Seq of TokenizedSentences from a Seq of Sentences.
This func generates a Seq of TokenizedSentences from a Seq of Sentences.
- sentences
to tag
- returns
Seq of TokenizedSentence objects
- Definition Classes
- TokenizerModel
- val targetPattern: Param[String]
pattern to grab from text as token candidates.
pattern to grab from text as token candidates. Defaults \\S+
- Definition Classes
- TokenizerModel
- def toString(): String
- Definition Classes
- Identifiable → AnyRef → Any
- final def transform(dataset: Dataset[_]): DataFrame
Given requirements are met, this applies ML transformation within a Pipeline or stand-alone Output annotation will be generated as a new column, previous annotations are still available separately metadata is built at schema level to record annotations structural information outside its content
Given requirements are met, this applies ML transformation within a Pipeline or stand-alone Output annotation will be generated as a new column, previous annotations are still available separately metadata is built at schema level to record annotations structural information outside its content
- dataset
Dataset[Row]
- Definition Classes
- AnnotatorModel → Transformer
- def transform(dataset: Dataset[_], paramMap: ParamMap): DataFrame
- Definition Classes
- Transformer
- Annotations
- @Since("2.0.0")
- def transform(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): DataFrame
- Definition Classes
- Transformer
- Annotations
- @varargs() @Since("2.0.0")
- final def transformSchema(schema: StructType): StructType
requirement for pipeline transformation validation.
requirement for pipeline transformation validation. It is called on fit()
- Definition Classes
- RawAnnotator → PipelineStage
- def transformSchema(schema: StructType, logging: Boolean): StructType
- Attributes
- protected
- Definition Classes
- PipelineStage
- Annotations
- @DeveloperApi()
- val uid: String
- Definition Classes
- ChunkTokenizerModel → TokenizerModel → Identifiable
- def validate(schema: StructType): Boolean
takes a Dataset and checks to see if all the required annotation types are present.
takes a Dataset and checks to see if all the required annotation types are present.
- schema
to be validated
- returns
True if all the required types are present, else false
- Attributes
- protected
- Definition Classes
- RawAnnotator
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- def wrapColumnMetadata(col: Column): Column
- Attributes
- protected
- Definition Classes
- RawAnnotator
- def write: MLWriter
- Definition Classes
- ParamsAndFeaturesWritable → DefaultParamsWritable → MLWritable
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)
Inherited from TokenizerModel
Inherited from HasSimpleAnnotate[TokenizerModel]
Inherited from AnnotatorModel[TokenizerModel]
Inherited from CanBeLazy
Inherited from RawAnnotator[TokenizerModel]
Inherited from HasOutputAnnotationCol
Inherited from HasInputAnnotationCols
Inherited from HasOutputAnnotatorType
Inherited from ParamsAndFeaturesWritable
Inherited from HasFeatures
Inherited from DefaultParamsWritable
Inherited from MLWritable
Inherited from Model[TokenizerModel]
Inherited from Transformer
Inherited from PipelineStage
Inherited from Logging
Inherited from Params
Inherited from Serializable
Inherited from Identifiable
Inherited from AnyRef
Inherited from Any
Parameters
A list of (hyper-)parameter keys this annotator can take. Users can set and get the parameter values through setters and getters, respectively.
Annotator types
Required input and expected output annotator types