object TransformerEncoder
- Companion:
- class
Type members
Value members
Concrete methods
def apply[S : Sc](numBlocks: Int, in: Int, attentionHiddenPerHeadDim: Int, attentionNumHeads: Int, mlpHiddenDim: Int, dropout: Double, padToken: Long, tOpt: STenOptions, linearized: Boolean): TransformerEncoder
Factory for the encoder module of transformer Does not include embedding and positional encoding
Factory for the encoder module of transformer Does not include embedding and positional encoding
Input is (data, tokens) where data is (batch, num tokens, in
dimension), double tensor tokens is (batch,num tokens) long tensor.
The sole purpose of tokens is to carry over the padding
- Value parameters:
- attentionHiddenPerHeadDim
size of hidden attention dimension of each attention head
- attentionNumHeads
number of attention heads
- dropout
dropout rate
- in
input dimension
- mlpHiddenDim
size of hidden dimension of the two layer perceptron
- numBlocks
number of transformer blocks to create
- out
output dimension
- padToken
pad token, (batch, seq) positions where
tokens==padTokenare ignored- tOpt
tensor options
- Returns:
a module