This is a Java implementation of a GPT3/4 tokenizer, loosely ported from Tiktoken with the help of ChatGPT. ...that all 3.5-turbo ...

JTokkit aims to be a fast and efficient tokenizer designed for use in natural language processing tasks using the OpenAI models.

Most Java programmers have used the java.util.StringTokenizer class at some time or another. It is a handy class that basically ...