public interface TextEncoder
Modifier and Type | Method and Description |
---|---|
io.spokestack.spokestack.nlu.tensorflow.EncodedTokens |
encode(String text)
Encode a raw string into identifiers for its constituent tokens.
|
int |
encodeSingle(String token)
Retrieves the identifier for the specified token without performing any
tokenization.
|
io.spokestack.spokestack.nlu.tensorflow.EncodedTokens encode(String text)
text
- The raw text to encode.EncodedTokens
object.int encodeSingle(String token)
If an unknown token is passed to this method, the implementation should map it to an identifier reserved for unknown tokens, and performance of the model will suffer accordingly.
token
- The token to encode.Copyright © 2020. All rights reserved.