Package io.codemodder.plugins.llm

Class Tokens

java.lang.Object
io.codemodder.plugins.llm.Tokens
public final class Tokens extends Object
A set of utilities around LLM tokens.

  • Method Summary

    Modifier and Type
    Method
    Description
    static int
    countTokens(List<com.theokanning.openai.completion.chat.ChatMessage> messages, int tokensPerMessage, com.knuddels.jtokkit.api.EncodingType encodingType)
    Estimates the number of tokens the messages will consume.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Method Details

    • countTokens

      public static int countTokens(List<com.theokanning.openai.completion.chat.ChatMessage> messages, int tokensPerMessage, com.knuddels.jtokkit.api.EncodingType encodingType)
      Estimates the number of tokens the messages will consume.

      This does not yet support estimating the number of tokens the functions will consume, since the unofficial solutions are brittle.

      We should be able to replace this with TikTokensUtil.tokens when the feature is released.

      Parameters:
      messages - The messages.
      tokensPerMessage - The number of tokens consumed per message by the given model.
      encodingType - The encoding type used by the model.
      Returns:
      The number of tokens.
      See Also: