Package io.codemodder.plugins.llm

Class Tokens

java.lang.Object

io.codemodder.plugins.llm.Tokens

public final class Tokens
extends Object

A set of utilities around LLM tokens.

Method Summary

Modifier and Type	Method	Description
static int	countTokens(List<String> messages, int tokensPerMessage, com.knuddels.jtokkit.api.EncodingType encodingType)	Estimates the number of tokens the messages will consume.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

countTokens

public static int countTokens(List<String> messages,
 int tokensPerMessage,
 com.knuddels.jtokkit.api.EncodingType encodingType)

Estimates the number of tokens the messages will consume.

This does not yet support estimating the number of tokens the functions will consume, since the unofficial solutions are brittle.

We should be able to replace this with TikTokensUtil.tokens when the feature is released.

Parameters:

messages - The messages.

tokensPerMessage - The number of tokens consumed per message by the given model.

encodingType - The encoding type used by the model.

Returns:

The number of tokens.

See Also:

• How to count tokens with tiktoken