BertPretrainInput

Input for BERT pretrain module

• Tokens: Long tensor of size (batch, sequence length). Sequence length includes cls and sep tokens. Values are tokens of the input vocabulary and 4 additional control tokens: cls, sep, pad, mask. First token must be cls.
• Segments: Long tensor of size (batch, sequence length). Values are segment tokens.
• Positions: Long tensor of size (batch, mask size (variable)). Values are indices in [0,sequence length) selecting masked sequence positions. They never select positions of cls, sep, pad.
• maxLength: 1D long tensor of size (sequence length). Values are in [0,sequence_length]. Tokens at positions higher or equal than the sequence length are ignored.