Class TokenTransformer
java.lang.Object
com.yahoo.searchlib.rankingexpression.transform.ExpressionTransformer<RankProfileTransformContext>
com.yahoo.schema.expressiontransforms.TokenTransformer
public class TokenTransformer
extends com.yahoo.searchlib.rankingexpression.transform.ExpressionTransformer<RankProfileTransformContext>
Convenience feature transforms for inputs to Transformer type models.
Replaces features of the form
tokenInputIds
tokenTypeIds
tokenAttentionMask
to tensor generation expressions that generate the required input.
In general, these models expect input of the form:
CLS + arg1 + SEP + arg2 + SEP + 0's
- Author:
- lesters
-
Constructor Summary
-
Method Summary
Modifier and TypeMethodDescriptionstatic com.yahoo.tensor.TensorType
createTensorType
(String featureName, com.yahoo.searchlib.rankingexpression.rule.ExpressionNode argument) com.yahoo.searchlib.rankingexpression.rule.ExpressionNode
transform
(com.yahoo.searchlib.rankingexpression.rule.ExpressionNode node, RankProfileTransformContext context) Methods inherited from class com.yahoo.searchlib.rankingexpression.transform.ExpressionTransformer
transform, transformChildren
-
Constructor Details
-
TokenTransformer
public TokenTransformer()
-
-
Method Details
-
transform
public com.yahoo.searchlib.rankingexpression.rule.ExpressionNode transform(com.yahoo.searchlib.rankingexpression.rule.ExpressionNode node, RankProfileTransformContext context) - Specified by:
transform
in classcom.yahoo.searchlib.rankingexpression.transform.ExpressionTransformer<RankProfileTransformContext>
-
createTensorType
public static com.yahoo.tensor.TensorType createTensorType(String featureName, com.yahoo.searchlib.rankingexpression.rule.ExpressionNode argument)
-