Class/Object

com.johnsnowlabs.nlp

Finisher

Related Docs: object Finisher | package nlp

Permalink

class Finisher extends Transformer with DefaultParamsWritable

Converts annotation results into a format that easier to use. It is useful to extract the results from Spark NLP Pipelines. The Finisher outputs annotation(s) values into String.

For more extended examples on document pre-processing see the Spark NLP Workshop.

Example

import spark.implicits._
import com.johnsnowlabs.nlp.pretrained.PretrainedPipeline
import com.johnsnowlabs.nlp.Finisher

val data = Seq((1, "New York and New Jersey aren't that far apart actually.")).toDF("id", "text")

// Extracts Named Entities amongst other things
val pipeline = PretrainedPipeline("explain_document_dl")

val finisher = new Finisher().setInputCols("entities").setOutputCols("output")
val explainResult = pipeline.transform(data)

explainResult.selectExpr("explode(entities)").show(false)
+------------------------------------------------------------------------------------------------------------------------------------------------------+
|entities                                                                                                                                              |
+------------------------------------------------------------------------------------------------------------------------------------------------------+
|[[chunk, 0, 7, New York, [entity -> LOC, sentence -> 0, chunk -> 0], []], [chunk, 13, 22, New Jersey, [entity -> LOC, sentence -> 0, chunk -> 1], []]]|
+------------------------------------------------------------------------------------------------------------------------------------------------------+

val result = finisher.transform(explainResult)
result.select("output").show(false)
+----------------------+
|output                |
+----------------------+
|[New York, New Jersey]|
+----------------------+
See also

EmbeddingsFinisher for finishing embeddings

Linear Supertypes
DefaultParamsWritable, MLWritable, Transformer, PipelineStage, Logging, Params, Serializable, Serializable, Identifiable, AnyRef, Any
Ordering
  1. Grouped
  2. Alphabetic
  3. By Inheritance
Inherited
  1. Finisher
  2. DefaultParamsWritable
  3. MLWritable
  4. Transformer
  5. PipelineStage
  6. Logging
  7. Params
  8. Serializable
  9. Serializable
  10. Identifiable
  11. AnyRef
  12. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new Finisher()

    Permalink

  2. new Finisher(uid: String)

    Permalink

    uid

    required uid for storing annotator to disk

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  3. final def $[T](param: Param[T]): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  4. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  5. val annotationSplitSymbol: Param[String]

    Permalink

    Character separating annotations (Default: @)

  6. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any

  7. val cleanAnnotations: BooleanParam

    Permalink

    Whether to remove annotation columns (Default: true)

  8. final def clear(param: Param[_]): Finisher.this.type

    Permalink
    Definition Classes
    Params

  9. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  10. def copy(extra: ParamMap): Transformer

    Permalink
    Definition Classes
    Finisher → Transformer → PipelineStage → Params

  11. def copyValues[T <: Params](to: T, extra: ParamMap): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  12. final def defaultCopy[T <: Params](extra: ParamMap): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  13. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  14. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any

  15. def explainParam(param: Param[_]): String

    Permalink
    Definition Classes
    Params

  16. def explainParams(): String

    Permalink
    Definition Classes
    Params

  17. final def extractParamMap(): ParamMap

    Permalink
    Definition Classes
    Params

  18. final def extractParamMap(extra: ParamMap): ParamMap

    Permalink
    Definition Classes
    Params

  19. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )

  20. final def get[T](param: Param[T]): Option[T]

    Permalink
    Definition Classes
    Params

  21. def getAnnotationSplitSymbol: String

    Permalink

    Character separating annotations (Default: #)

  22. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any

  23. def getCleanAnnotations: Boolean

    Permalink

    Whether to remove annotation columns (Default: true)

  24. final def getDefault[T](param: Param[T]): Option[T]

    Permalink
    Definition Classes
    Params

  25. def getIncludeMetadata: Boolean

    Permalink

    Annotation metadata format (Default: false)

  26. def getInputCols: Array[String]

    Permalink

    Name of finisher output cols

  27. final def getOrDefault[T](param: Param[T]): T

    Permalink
    Definition Classes
    Params

  28. def getOutputAsArray: Boolean

    Permalink

    Finisher generates an Array with the results instead of string (Default: true)

  29. def getOutputCols: Array[String]

    Permalink

    Name of input annotation cols

  30. def getParam(paramName: String): Param[Any]

    Permalink
    Definition Classes
    Params

  31. def getValueSplitSymbol: String

    Permalink

    Character separating annotations (Default: #)

  32. final def hasDefault[T](param: Param[T]): Boolean

    Permalink
    Definition Classes
    Params

  33. def hasParam(paramName: String): Boolean

    Permalink
    Definition Classes
    Params

  34. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any

  35. val includeMetadata: BooleanParam

    Permalink

    Annotation metadata format (Default: false)

  36. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  37. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  38. val inputCols: StringArrayParam

    Permalink

    Name of input annotation cols

  39. final def isDefined(param: Param[_]): Boolean

    Permalink
    Definition Classes
    Params

  40. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any

  41. final def isSet(param: Param[_]): Boolean

    Permalink
    Definition Classes
    Params

  42. def isTraceEnabled(): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  43. def log: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  44. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  45. def logDebug(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  46. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  47. def logError(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  48. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  49. def logInfo(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  50. def logName: String

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  51. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  52. def logTrace(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  53. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  54. def logWarning(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging

  55. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef

  56. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef

  57. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef

  58. val outputAsArray: BooleanParam

    Permalink

    Finisher generates an Array with the results instead of string (Default: true)

  59. val outputCols: StringArrayParam

    Permalink

    Name of finisher output cols

  60. lazy val params: Array[Param[_]]

    Permalink
    Definition Classes
    Params

  61. val parseEmbeddingsVectors: BooleanParam

    Permalink

    Whether to include embeddings vectors in the process (Default: false)

  62. def save(path: String): Unit

    Permalink
    Definition Classes
    MLWritable
    Annotations
    @Since( "1.6.0" ) @throws( ... )

  63. final def set(paramPair: ParamPair[_]): Finisher.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  64. final def set(param: String, value: Any): Finisher.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  65. final def set[T](param: Param[T], value: T): Finisher.this.type

    Permalink
    Definition Classes
    Params

  66. def setAnnotationSplitSymbol(value: String): Finisher.this.type

    Permalink

    Character separating annotations (Default: #)

  67. def setCleanAnnotations(value: Boolean): Finisher.this.type

    Permalink

    Whether to remove annotation columns (Default: true)

  68. final def setDefault(paramPairs: ParamPair[_]*): Finisher.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  69. final def setDefault[T](param: Param[T], value: T): Finisher.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params

  70. def setIncludeMetadata(value: Boolean): Finisher.this.type

    Permalink

    Annotation metadata format (Default: false)

  71. def setInputCols(value: String*): Finisher.this.type

    Permalink

    Name of input annotation cols

  72. def setInputCols(value: Array[String]): Finisher.this.type

    Permalink

    Name of input annotation cols

  73. def setOutputAsArray(value: Boolean): Finisher.this.type

    Permalink

    Finisher generates an Array with the results instead of string (Default: true)

  74. def setOutputCols(value: String*): Finisher.this.type

    Permalink

    Name of finisher output cols

  75. def setOutputCols(value: Array[String]): Finisher.this.type

    Permalink

    Name of finisher output cols

  76. def setValueSplitSymbol(value: String): Finisher.this.type

    Permalink

    Character separating annotations (Default: #)

  77. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef

  78. def toString(): String

    Permalink
    Definition Classes
    Identifiable → AnyRef → Any

  79. def transform(dataset: Dataset[_]): Dataset[Row]

    Permalink
    Definition Classes
    Finisher → Transformer

  80. def transform(dataset: Dataset[_], paramMap: ParamMap): DataFrame

    Permalink
    Definition Classes
    Transformer
    Annotations
    @Since( "2.0.0" )

  81. def transform(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): DataFrame

    Permalink
    Definition Classes
    Transformer
    Annotations
    @Since( "2.0.0" ) @varargs()

  82. def transformSchema(schema: StructType): StructType

    Permalink
    Definition Classes
    Finisher → PipelineStage

  83. def transformSchema(schema: StructType, logging: Boolean): StructType

    Permalink
    Attributes
    protected
    Definition Classes
    PipelineStage
    Annotations
    @DeveloperApi()

  84. val uid: String

    Permalink

    required uid for storing annotator to disk

    required uid for storing annotator to disk

    Definition Classes
    Finisher → Identifiable

  85. val valueSplitSymbol: Param[String]

    Permalink

    Character separating annotations (Default: #)

  86. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  87. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  88. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

  89. def write: MLWriter

    Permalink
    Definition Classes
    DefaultParamsWritable → MLWritable

Inherited from DefaultParamsWritable

Inherited from MLWritable

Inherited from Transformer

Inherited from PipelineStage

Inherited from Logging

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

Parameters

A list of (hyper-)parameter keys this annotator can take. Users can set and get the parameter values through setters and getters, respectively.

Members

Parameter setters

Parameter getters