Package io.codemodder.plugins.llm

Class Tokens

java.lang.Object

io.codemodder.plugins.llm.Tokens

public final class Tokens
extends Object

A set of utilities around LLM tokens.

Method Summary

Modifier and Type	Method	Description
static int	countTokens(List<com.theokanning.openai.completion.chat.ChatMessage> messages)	Estimates the number of tokens the messages will consume when passed to the gpt-3.5-turbo-0613 or gpt-4-0613 models.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

countTokens

public static int countTokens(List<com.theokanning.openai.completion.chat.ChatMessage> messages)

Estimates the number of tokens the messages will consume when passed to the gpt-3.5-turbo-0613 or gpt-4-0613 models.

This does not yet support estimating the number of tokens the functions will consume, since the unofficial solutions are brittle.

We should be able to replace this with TikTokensUtil.tokens when the feature is released.

Parameters:

messages - The messages.

Returns:

The number of tokens.

See Also:

• How to count tokens with tiktoken