com.johnsnowlabs.nlp.annotators.sentence_detector_dl
SentenceDetectorDLApproach
Related Doc:
package sentence_detector_dl
com.johnsnowlabs.nlp.annotators.sentence_detector_dl
Trains TensorFlow model for multi-class text classification
Trains TensorFlow model for multi-class text classification
Maximum number of epochs to train (Default: 5)
A flag indicating whether to split sentences into different Dataset rows.
A flag indicating whether to split sentences into different Dataset rows. Useful for higher parallelism in
fat rows (Default: false)
Maximum number of epochs to train (Default: 5)
Whether to split sentences into different Dataset rows.
Whether to split sentences into different Dataset rows. Useful for higher parallelism in fat rows. Defaults to false.
Get impossible penultimates
input annotations columns currently used
Get model architecture
Gets annotation column name going to generate
Gets annotation column name going to generate
Get output logs path
Choose the proportion of training dataset to be validated against the model on each Epoch.
Choose the proportion of training dataset to be validated against the model on each Epoch. The value should be between 0.0 and 1.0 and by default it is 0.0 and off.
Impossible penultimates, which should not be split on Default:
Impossible penultimates, which should not be split on Default:
Array( "Bros", "No", "al", "vs", "etc", "Fig", "Dr", "Prof", "PhD", "MD", "Co", "Corp", "Inc", "bros", "VS", "Vs", "ETC", "fig", "dr", "prof", "PHD", "phd", "md", "co", "corp", "inc", "Jan", "Feb", "Mar", "Apr", "Jul", "Aug", "Sep", "Sept", "Oct", "Nov", "Dec", "St", "st", "AM", "PM", "am", "pm", "e.g", "f.e", "i.e" )
Input annotator type : DOCUMENT
Input annotator type : DOCUMENT
columns that contain annotations necessary to run this annotator AnnotatorType is used both as input and output columns if not specified
columns that contain annotations necessary to run this annotator AnnotatorType is used both as input and output columns if not specified
Model architecture (Default: "cnn")
Output annotator type : DOCUMENT
Output annotator type : DOCUMENT
Path to folder to output logs (Default: "")
If no path is specified, no logs are generated
Maximum number of epochs to train (Default: 5)
Whether to split sentences into different Dataset rows.
Whether to split sentences into different Dataset rows. Useful for higher parallelism in fat rows. Defaults to false.
Set impossible penultimates
Overrides required annotators column if different than default
Overrides required annotators column if different than default
Set architecture
Overrides annotation column name when transforming
Overrides annotation column name when transforming
Set the output log path
Choose the proportion of training dataset to be validated against the model on each Epoch.
Choose the proportion of training dataset to be validated against the model on each Epoch. The value should be between 0.0 and 1.0 and by default it is 0.0 and off.
requirement for pipeline transformation validation.
requirement for pipeline transformation validation. It is called on fit()
required uid for storing annotator to disk
required uid for storing annotator to disk
takes a Dataset and checks to see if all the required annotation types are present.
takes a Dataset and checks to see if all the required annotation types are present.
to be validated
True if all the required types are present, else false
Choose the proportion of training dataset to be validated against the model on each Epoch (Default: 0.0f).
Choose the proportion of training dataset to be validated against the model on each Epoch (Default: 0.0f).
The value should be between 0.0 and 1.0 and by default it is 0.0 and off.
A list of (hyper-)parameter keys this annotator can take. Users can set and get the parameter values through setters and getters, respectively.
Required input and expected output annotator types
Trains an annotator that detects sentence boundaries using a deep learning approach.
For pretrained models see SentenceDetectorDLModel.
Currently, only the CNN model is supported for training, but in the future the architecture of the model can be set with
setModelArchitecture.The default model
"cnn"is based on the paper Deep-EOS: General-Purpose Neural Networks for Sentence Boundary Detection (2020, Stefan Schweter, Sajawel Ahmed) using a CNN architecture. We also modified the original implementation a little bit to cover broken sentences and some impossible end of line chars.Each extracted sentence can be returned in an Array or exploded to separate rows, if
explodeSentencesis set totrue.For extended examples of usage, see the SentenceDetectorDLSpec.
Example
The training process needs data, where each data point is a sentence.
In this example the
train.txtfile has the form ofwhere each line is one sentence. Training can then be started like so:
SentenceDetector for non deep learning extraction
SentenceDetectorDLModel for pretrained models