c
com.johnsnowlabs.nlp.annotators.classifier.dl
DistilBertForMultipleChoice
Companion object DistilBertForMultipleChoice
class DistilBertForMultipleChoice extends AnnotatorModel[DistilBertForMultipleChoice] with HasBatchedAnnotate[DistilBertForMultipleChoice] with WriteOnnxModel with WriteOpenvinoModel with HasCaseSensitiveProperties with HasEngine
Linear Supertypes
Ordering
- Grouped
- Alphabetic
- By Inheritance
Inherited
- DistilBertForMultipleChoice
- HasEngine
- HasCaseSensitiveProperties
- WriteOpenvinoModel
- WriteOnnxModel
- HasBatchedAnnotate
- AnnotatorModel
- CanBeLazy
- RawAnnotator
- HasOutputAnnotationCol
- HasInputAnnotationCols
- HasOutputAnnotatorType
- ParamsAndFeaturesWritable
- HasFeatures
- DefaultParamsWritable
- MLWritable
- Model
- Transformer
- PipelineStage
- Logging
- Params
- Serializable
- Identifiable
- AnyRef
- Any
- Hide All
- Show All
Visibility
- Public
- Protected
Instance Constructors
Type Members
- type AnnotationContent = Seq[Row]
internal types to show Rows as a relevant StructType Should be deleted once Spark releases UserDefinedTypes to @developerAPI
internal types to show Rows as a relevant StructType Should be deleted once Spark releases UserDefinedTypes to @developerAPI
- Attributes
- protected
- Definition Classes
- AnnotatorModel
- type AnnotatorType = String
- Definition Classes
- HasOutputAnnotatorType
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def $[T](param: Param[T]): T
- Attributes
- protected
- Definition Classes
- Params
- def $$[T](feature: StructFeature[T]): T
- Attributes
- protected
- Definition Classes
- HasFeatures
- def $$[K, V](feature: MapFeature[K, V]): Map[K, V]
- Attributes
- protected
- Definition Classes
- HasFeatures
- def $$[T](feature: SetFeature[T]): Set[T]
- Attributes
- protected
- Definition Classes
- HasFeatures
- def $$[T](feature: ArrayFeature[T]): Array[T]
- Attributes
- protected
- Definition Classes
- HasFeatures
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- def _transform(dataset: Dataset[_], recursivePipeline: Option[PipelineModel]): DataFrame
- Attributes
- protected
- Definition Classes
- AnnotatorModel
- def afterAnnotate(dataset: DataFrame): DataFrame
- Attributes
- protected
- Definition Classes
- AnnotatorModel
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def batchAnnotate(batchedAnnotations: Seq[Array[Annotation]]): Seq[Seq[Annotation]]
takes a document and annotations and produces new annotations of this annotator's annotation type
takes a document and annotations and produces new annotations of this annotator's annotation type
- batchedAnnotations
Annotations in batches that correspond to inputAnnotationCols generated by previous annotators if any
- returns
any number of annotations processed for every batch of input annotations. Not necessary one to one relationship IMPORTANT: !MUST! return sequences of equal lengths !! IMPORTANT: !MUST! return sentences that belong to the same original row !! (challenging)
- Definition Classes
- DistilBertForMultipleChoice → HasBatchedAnnotate
- def batchProcess(rows: Iterator[_]): Iterator[Row]
- Definition Classes
- HasBatchedAnnotate
- val batchSize: IntParam
Size of every batch (Default depends on model).
Size of every batch (Default depends on model).
- Definition Classes
- HasBatchedAnnotate
- def beforeAnnotate(dataset: Dataset[_]): Dataset[_]
- Attributes
- protected
- Definition Classes
- AnnotatorModel
- val caseSensitive: BooleanParam
Whether to ignore case in index lookups (Default depends on model)
Whether to ignore case in index lookups (Default depends on model)
- Definition Classes
- HasCaseSensitiveProperties
- final def checkSchema(schema: StructType, inputAnnotatorType: String): Boolean
- Attributes
- protected
- Definition Classes
- HasInputAnnotationCols
- val choicesDelimiter: Param[String]
- final def clear(param: Param[_]): DistilBertForMultipleChoice.this.type
- Definition Classes
- Params
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @HotSpotIntrinsicCandidate() @native()
- def copy(extra: ParamMap): DistilBertForMultipleChoice
requirement for annotators copies
requirement for annotators copies
- Definition Classes
- RawAnnotator → Model → Transformer → PipelineStage → Params
- def copyValues[T <: Params](to: T, extra: ParamMap): T
- Attributes
- protected
- Definition Classes
- Params
- final def defaultCopy[T <: Params](extra: ParamMap): T
- Attributes
- protected
- Definition Classes
- Params
- val engine: Param[String]
This param is set internally once via loadSavedModel.
This param is set internally once via loadSavedModel. That's why there is no setter
- Definition Classes
- HasEngine
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def explainParam(param: Param[_]): String
- Definition Classes
- Params
- def explainParams(): String
- Definition Classes
- Params
- def extraValidate(structType: StructType): Boolean
- Attributes
- protected
- Definition Classes
- RawAnnotator
- def extraValidateMsg: String
Override for additional custom schema checks
Override for additional custom schema checks
- Attributes
- protected
- Definition Classes
- RawAnnotator
- final def extractParamMap(): ParamMap
- Definition Classes
- Params
- final def extractParamMap(extra: ParamMap): ParamMap
- Definition Classes
- Params
- val features: ArrayBuffer[Feature[_, _, _]]
- Definition Classes
- HasFeatures
- def get[T](feature: StructFeature[T]): Option[T]
- Attributes
- protected
- Definition Classes
- HasFeatures
- def get[K, V](feature: MapFeature[K, V]): Option[Map[K, V]]
- Attributes
- protected
- Definition Classes
- HasFeatures
- def get[T](feature: SetFeature[T]): Option[Set[T]]
- Attributes
- protected
- Definition Classes
- HasFeatures
- def get[T](feature: ArrayFeature[T]): Option[Array[T]]
- Attributes
- protected
- Definition Classes
- HasFeatures
- final def get[T](param: Param[T]): Option[T]
- Definition Classes
- Params
- def getBatchSize: Int
Size of every batch.
Size of every batch.
- Definition Classes
- HasBatchedAnnotate
- def getCaseSensitive: Boolean
- Definition Classes
- HasCaseSensitiveProperties
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @HotSpotIntrinsicCandidate() @native()
- final def getDefault[T](param: Param[T]): Option[T]
- Definition Classes
- Params
- def getEngine: String
- Definition Classes
- HasEngine
- def getInputCols: Array[String]
- returns
input annotations columns currently used
- Definition Classes
- HasInputAnnotationCols
- def getLazyAnnotator: Boolean
- Definition Classes
- CanBeLazy
- def getModelIfNotSet: DistilBertClassification
- final def getOrDefault[T](param: Param[T]): T
- Definition Classes
- Params
- final def getOutputCol: String
Gets annotation column name going to generate
Gets annotation column name going to generate
- Definition Classes
- HasOutputAnnotationCol
- def getParam(paramName: String): Param[Any]
- Definition Classes
- Params
- final def hasDefault[T](param: Param[T]): Boolean
- Definition Classes
- Params
- def hasParam(paramName: String): Boolean
- Definition Classes
- Params
- def hasParent: Boolean
- Definition Classes
- Model
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @HotSpotIntrinsicCandidate() @native()
- def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
- Attributes
- protected
- Definition Classes
- Logging
- def initializeLogIfNecessary(isInterpreter: Boolean): Unit
- Attributes
- protected
- Definition Classes
- Logging
- val inputAnnotatorTypes: Array[AnnotatorType]
Annotator reference id.
Annotator reference id. Used to identify elements in metadata or to refer to this annotator type
- Definition Classes
- DistilBertForMultipleChoice → HasInputAnnotationCols
- final val inputCols: StringArrayParam
columns that contain annotations necessary to run this annotator AnnotatorType is used both as input and output columns if not specified
columns that contain annotations necessary to run this annotator AnnotatorType is used both as input and output columns if not specified
- Attributes
- protected
- Definition Classes
- HasInputAnnotationCols
- final def isDefined(param: Param[_]): Boolean
- Definition Classes
- Params
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def isSet(param: Param[_]): Boolean
- Definition Classes
- Params
- def isTraceEnabled(): Boolean
- Attributes
- protected
- Definition Classes
- Logging
- val lazyAnnotator: BooleanParam
- Definition Classes
- CanBeLazy
- def log: Logger
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logDebug(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logError(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logInfo(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logName: String
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logTrace(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(msg: => String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def logWarning(msg: => String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- val maxSentenceLength: IntParam
Max sentence length to process (Default:
512) - def msgHelper(schema: StructType): String
- Attributes
- protected
- Definition Classes
- HasInputAnnotationCols
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @HotSpotIntrinsicCandidate() @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @HotSpotIntrinsicCandidate() @native()
- def onWrite(path: String, spark: SparkSession): Unit
- Definition Classes
- DistilBertForMultipleChoice → ParamsAndFeaturesWritable
- val optionalInputAnnotatorTypes: Array[String]
- Definition Classes
- HasInputAnnotationCols
- val outputAnnotatorType: AnnotatorType
- Definition Classes
- DistilBertForMultipleChoice → HasOutputAnnotatorType
- final val outputCol: Param[String]
- Attributes
- protected
- Definition Classes
- HasOutputAnnotationCol
- lazy val params: Array[Param[_]]
- Definition Classes
- Params
- var parent: Estimator[DistilBertForMultipleChoice]
- Definition Classes
- Model
- def save(path: String): Unit
- Definition Classes
- MLWritable
- Annotations
- @throws("If the input path already exists but overwrite is not enabled.") @Since("1.6.0")
- def sentenceEndTokenId: Int
- def sentenceStartTokenId: Int
- def set[T](feature: StructFeature[T], value: T): DistilBertForMultipleChoice.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
- def set[K, V](feature: MapFeature[K, V], value: Map[K, V]): DistilBertForMultipleChoice.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
- def set[T](feature: SetFeature[T], value: Set[T]): DistilBertForMultipleChoice.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
- def set[T](feature: ArrayFeature[T], value: Array[T]): DistilBertForMultipleChoice.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
- final def set(paramPair: ParamPair[_]): DistilBertForMultipleChoice.this.type
- Attributes
- protected
- Definition Classes
- Params
- final def set(param: String, value: Any): DistilBertForMultipleChoice.this.type
- Attributes
- protected
- Definition Classes
- Params
- final def set[T](param: Param[T], value: T): DistilBertForMultipleChoice.this.type
- Definition Classes
- Params
- def setBatchSize(size: Int): DistilBertForMultipleChoice.this.type
Size of every batch.
Size of every batch.
- Definition Classes
- HasBatchedAnnotate
- def setCaseSensitive(value: Boolean): DistilBertForMultipleChoice.this.type
Whether to lowercase tokens or not (Default:
true).Whether to lowercase tokens or not (Default:
true).- Definition Classes
- DistilBertForMultipleChoice → HasCaseSensitiveProperties
- def setChoicesDelimiter(value: String): DistilBertForMultipleChoice.this.type
- def setDefault[T](feature: StructFeature[T], value: () => T): DistilBertForMultipleChoice.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
- def setDefault[K, V](feature: MapFeature[K, V], value: () => Map[K, V]): DistilBertForMultipleChoice.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
- def setDefault[T](feature: SetFeature[T], value: () => Set[T]): DistilBertForMultipleChoice.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
- def setDefault[T](feature: ArrayFeature[T], value: () => Array[T]): DistilBertForMultipleChoice.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
- final def setDefault(paramPairs: ParamPair[_]*): DistilBertForMultipleChoice.this.type
- Attributes
- protected
- Definition Classes
- Params
- final def setDefault[T](param: Param[T], value: T): DistilBertForMultipleChoice.this.type
- Attributes
- protected[org.apache.spark.ml]
- Definition Classes
- Params
- final def setInputCols(value: String*): DistilBertForMultipleChoice.this.type
- Definition Classes
- HasInputAnnotationCols
- def setInputCols(value: Array[String]): DistilBertForMultipleChoice.this.type
Overrides required annotators column if different than default
Overrides required annotators column if different than default
- Definition Classes
- HasInputAnnotationCols
- def setLazyAnnotator(value: Boolean): DistilBertForMultipleChoice.this.type
- Definition Classes
- CanBeLazy
- def setMaxSentenceLength(value: Int): DistilBertForMultipleChoice.this.type
- def setModelIfNotSet(spark: SparkSession, tensorflowWrapper: Option[TensorflowWrapper], onnxWrapper: Option[OnnxWrapper], openvinoWrapper: Option[OpenvinoWrapper]): DistilBertForMultipleChoice
- final def setOutputCol(value: String): DistilBertForMultipleChoice.this.type
Overrides annotation column name when transforming
Overrides annotation column name when transforming
- Definition Classes
- HasOutputAnnotationCol
- def setParent(parent: Estimator[DistilBertForMultipleChoice]): DistilBertForMultipleChoice
- Definition Classes
- Model
- def setVocabulary(value: Map[String, Int]): DistilBertForMultipleChoice.this.type
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- Identifiable → AnyRef → Any
- final def transform(dataset: Dataset[_]): DataFrame
Given requirements are met, this applies ML transformation within a Pipeline or stand-alone Output annotation will be generated as a new column, previous annotations are still available separately metadata is built at schema level to record annotations structural information outside its content
Given requirements are met, this applies ML transformation within a Pipeline or stand-alone Output annotation will be generated as a new column, previous annotations are still available separately metadata is built at schema level to record annotations structural information outside its content
- dataset
Dataset[Row]
- Definition Classes
- AnnotatorModel → Transformer
- def transform(dataset: Dataset[_], paramMap: ParamMap): DataFrame
- Definition Classes
- Transformer
- Annotations
- @Since("2.0.0")
- def transform(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): DataFrame
- Definition Classes
- Transformer
- Annotations
- @varargs() @Since("2.0.0")
- final def transformSchema(schema: StructType): StructType
requirement for pipeline transformation validation.
requirement for pipeline transformation validation. It is called on fit()
- Definition Classes
- RawAnnotator → PipelineStage
- def transformSchema(schema: StructType, logging: Boolean): StructType
- Attributes
- protected
- Definition Classes
- PipelineStage
- Annotations
- @DeveloperApi()
- val uid: String
- Definition Classes
- DistilBertForMultipleChoice → Identifiable
- def validate(schema: StructType): Boolean
takes a Dataset and checks to see if all the required annotation types are present.
takes a Dataset and checks to see if all the required annotation types are present.
- schema
to be validated
- returns
True if all the required types are present, else false
- Attributes
- protected
- Definition Classes
- RawAnnotator
- val vocabulary: MapFeature[String, Int]
Vocabulary used to encode the words to ids with WordPieceEncoder
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- def wrapColumnMetadata(col: Column): Column
- Attributes
- protected
- Definition Classes
- RawAnnotator
- def write: MLWriter
- Definition Classes
- ParamsAndFeaturesWritable → DefaultParamsWritable → MLWritable
- def writeOnnxModel(path: String, spark: SparkSession, onnxWrapper: OnnxWrapper, suffix: String, fileName: String): Unit
- Definition Classes
- WriteOnnxModel
- def writeOnnxModels(path: String, spark: SparkSession, onnxWrappersWithNames: Seq[(OnnxWrapper, String)], suffix: String): Unit
- Definition Classes
- WriteOnnxModel
- def writeOpenvinoModel(path: String, spark: SparkSession, openvinoWrapper: OpenvinoWrapper, suffix: String, fileName: String): Unit
- Definition Classes
- WriteOpenvinoModel
- def writeOpenvinoModels(path: String, spark: SparkSession, ovWrappersWithNames: Seq[(OpenvinoWrapper, String)], suffix: String): Unit
- Definition Classes
- WriteOpenvinoModel
Deprecated Value Members
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable]) @Deprecated
- Deprecated
(Since version 9)