Important: This documentation covers Yarn 1 (Classic).

For Yarn 2+ docs and migration guide, see yarnpkg.com.

keywords: Text Generation

found 34 packages in 2ms

gpt-tokenizer1.1mMIT3.2.0

A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models

niieaniabout 9 hours agoBPE, encoder, decoder, tokenizer

npmGitHub

openai-gpt-token-counter32.4kISC1.1.2

Count the number of OpenAI tokens in a string. Supports all OpenAI Text models including GPT-5, GPT-4, GPT-3.5-turbo, and specialized models

codergautam4 days agoopenai, gpt, gpt3, openai tokens

npmGitHub

gptoken3.5kMIT0.1.0

Javascript BPE Encoder Decoder for GPT-2 / GPT-3. The "gpt-3-encoder" module provides functions for encoding and decoding text using the Byte Pair Encoding (BPE) algorithm. It can be used to process text data for input into machine learning models, or to