object BertLoss
- Companion:
- class
Type members
Value members
Concrete methods
def apply[S : Sc](maxLength: Int, vocabularySize: Int, segmentVocabularySize: Int, mlmHiddenDim: Int, wholeStentenceHiddenDim: Int, numBlocks: Int, embeddingDim: Int, attentionHiddenPerHeadDim: Int, attentionNumHeads: Int, bertEncoderMlpHiddenDim: Int, dropout: Double, padToken: Long, tOpt: STenOptions, linearized: Boolean, positionEmbedding: Option[STen]): BertLoss
Allocate Bert module
Allocate Bert module
- Value parameters:
- bertEncoderMlpHiddenDim
Hidden dimension within transformer blocks
- embeddingDim
Width of the initial embedding dimension, as well as the output width of the feed forward network in each transformer block
- linearized
Whether to use linearized self attention
- maxLength
Total sequence length including cls, sep and potential pad tokens
- mlmHiddenDim
Hidden dimension of the masked language model decoder
- numBlocks
Number of transformer blocks
- padToken
pad will be ignored
- segmentVocabularySize
Vocabulary size of the segment features
- vocabularySize
Total vocabulary size including cls, sep, pad, mask tokens
- wholeStentenceHiddenDim
Hidden dimension of the whole sentence task decoder