com.johnsnowlabs.nlp.annotators.ner.dl
NerDLApproach
Related Docs:
object NerDLApproach
| package dl
com.johnsnowlabs.nlp.annotators.ner.dl
Batch size
ConfigProto from tensorflow, serialized into byte array.
ConfigProto from tensorflow, serialized into byte array. Get with config_proto.SerializeToString()
Trains Tensorflow based Char-CNN-BLSTM model
Trains Tensorflow based Char-CNN-BLSTM model
"Dropout coefficient
Whether to output to annotators log folder
Entities to recognize
Entities to recognize
Whether logs for validation to be extended: it displays time and evaluation of each label.
Whether logs for validation to be extended: it displays time and evaluation of each label. Default is false.
Batch size
ConfigProto from tensorflow, serialized into byte array.
ConfigProto from tensorflow, serialized into byte array. Get with config_proto.SerializeToString()
Dropout coefficient
Whether to output to annotators log folder
whether to include confidence scores in annotation metadata
input annotations columns currently used
Learning Rate
Maximum number of epochs to train
Maximum number of epochs to train
Minimum number of epochs to train
Minimum number of epochs to train
Gets annotation column name going to generate
Gets annotation column name going to generate
Learning rate decay coefficient.
Learning rate decay coefficient. Real Learning Rage = lr / (1 + po * epoch)
Random seed
Random seed
Whether to use contrib LSTM Cells.
Whether to use contrib LSTM Cells. Not compatible with Windows. Might slightly improve accuracy.
Choose the proportion of training dataset to be validated against the model on each Epoch.
Choose the proportion of training dataset to be validated against the model on each Epoch. The value should be between 0.0 and 1.0 and by default it is 0.0 and off.
Level of verbosity during training
Level of verbosity during training
Folder path that contain external graph files
val includeConfidence = new BooleanParam(this, "includeConfidence", "Whether to include confidence scores in annotation metadata")
Input annotator types : DOCUMENT, TOKEN, WORD_EMBEDDINGS
Input annotator types : DOCUMENT, TOKEN, WORD_EMBEDDINGS
columns that contain annotations necessary to run this annotator AnnotatorType is used both as input and output columns if not specified
columns that contain annotations necessary to run this annotator AnnotatorType is used both as input and output columns if not specified
Column with label per each token
Column with label per each token
Learning Rate
Maximum number of epochs to train
Maximum number of epochs to train
Minimum number of epochs to train
Minimum number of epochs to train
Input annotator types : NAMED_ENTITY
Input annotator types : NAMED_ENTITY
Learning rate decay coefficient.
Learning rate decay coefficient. Real Learning Rage = lr / (1 + po * epoch)
Random seed
Random seed
Batch size
ConfigProto from tensorflow, serialized into byte array.
ConfigProto from tensorflow, serialized into byte array. Get with config_proto.SerializeToString()
Dropout coefficient
Whether to output to annotators log folder
Entities to recognize
Entities to recognize
Whether logs for validation to be extended: it displays time and evaluation of each label.
Whether logs for validation to be extended: it displays time and evaluation of each label. Default is false.
Folder path that contain external graph files
Whether to include confidence scores in annotation metadata
Overrides required annotators column if different than default
Overrides required annotators column if different than default
Column with label per each token
Column with label per each token
Learning Rate
Maximum number of epochs to train
Maximum number of epochs to train
Minimum number of epochs to train
Minimum number of epochs to train
Overrides annotation column name when transforming
Overrides annotation column name when transforming
Learning rate decay coefficient.
Learning rate decay coefficient. Real Learning Rage = lr / (1 + po * epoch)
Random seed
Random seed
Path to test dataset.
Path to test dataset. If set used to calculate statistic on it during training.
Path to test dataset.
Path to test dataset. If set used to calculate statistic on it during training.
Whether to use contrib LSTM Cells.
Whether to use contrib LSTM Cells. Not compatible with Windows. Might slightly improve accuracy.
Choose the proportion of training dataset to be validated against the model on each Epoch.
Choose the proportion of training dataset to be validated against the model on each Epoch. The value should be between 0.0 and 1.0 and by default it is 0.0 and off.
Level of verbosity during training
Level of verbosity during training
Level of verbosity during training
Level of verbosity during training
val testDataset = new ExternalResourceParam(this, "testDataset", "Path to test dataset.
val testDataset = new ExternalResourceParam(this, "testDataset", "Path to test dataset. If set used to calculate statistic on it during training.")
requirement for pipeline transformation validation.
requirement for pipeline transformation validation. It is called on fit()
whether to use contrib LSTM Cells.
whether to use contrib LSTM Cells. Not compatible with Windows. Might slightly improve accuracy.
takes a Dataset and checks to see if all the required annotation types are present.
takes a Dataset and checks to see if all the required annotation types are present.
to be validated
True if all the required types are present, else false
Choose the proportion of training dataset to be validated against the model on each Epoch.
Choose the proportion of training dataset to be validated against the model on each Epoch. The value should be between 0.0 and 1.0 and by default it is 0.0 and off.
Level of verbosity during training
Level of verbosity during training
Required input and expected output annotator types
This Named Entity recognition annotator allows to train generic NER model based on Neural Networks. Its train data (train_ner) is either a labeled or an external CoNLL 2003 IOB based spark dataset with Annotations columns. Also the user has to provide word embeddings annotation column. Neural Network architecture is Char CNNs - BiLSTM - CRF that achieves state-of-the-art in most datasets.
See https://github.com/JohnSnowLabs/spark-nlp/tree/master/src/test/scala/com/johnsnowlabs/nlp/annotators/ner/dl for further reference on how to use this API.