Package com.yahoo.searchdefinition.expressiontransforms

Class TokenTransformer

java.lang.Object

com.yahoo.searchlib.rankingexpression.transform.ExpressionTransformer<RankProfileTransformContext>

com.yahoo.searchdefinition.expressiontransforms.TokenTransformer

public class TokenTransformer
extends com.yahoo.searchlib.rankingexpression.transform.ExpressionTransformer<RankProfileTransformContext>

Convenience feature transforms for inputs to Transformer type models. Replaces features of the form tokenInputIds tokenTypeIds tokenAttentionMask to tensor generation expressions that generate the required input. In general, these models expect input of the form: CLS + arg1 + SEP + arg2 + SEP + 0's

Author:

lesters

Constructor Summary

Constructors

Constructor	Description
TokenTransformer()

Method Summary

Modifier and Type	Method	Description
static com.yahoo.tensor.TensorType	createTensorType(String featureName, com.yahoo.searchlib.rankingexpression.rule.ExpressionNode argument)
com.yahoo.searchlib.rankingexpression.rule.ExpressionNode	transform(com.yahoo.searchlib.rankingexpression.rule.ExpressionNode node, RankProfileTransformContext context)

Methods inherited from class com.yahoo.searchlib.rankingexpression.transform.ExpressionTransformer

transform, transformChildren

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TokenTransformer

public TokenTransformer()

Method Detail

transform

public com.yahoo.searchlib.rankingexpression.rule.ExpressionNode transform(com.yahoo.searchlib.rankingexpression.rule.ExpressionNode node,
                                                                           RankProfileTransformContext context)

Specified by:

transform in class com.yahoo.searchlib.rankingexpression.transform.ExpressionTransformer<RankProfileTransformContext>

createTensorType

public static com.yahoo.tensor.TensorType createTensorType(String featureName,
                                                           com.yahoo.searchlib.rankingexpression.rule.ExpressionNode argument)