Important: This documentation covers Yarn 1 (Classic).

For Yarn 2+ docs and migration guide, see yarnpkg.com.

keywords: BPE

found 19 packages in 2ms

gpt-tokenizer1.1mMIT3.4.0

A pure JavaScript implementation of a BPE tokenizer (Encoder/Decoder) for GPT-2 / GPT-3 / GPT-4 and other OpenAI models

niieani3 months agoBPE, encoder, decoder, tokenizer

gptoken2.8kMIT0.1.0

Javascript BPE Encoder Decoder for GPT-2 / GPT-3. The "gpt-3-encoder" module provides functions for encoding and decoding text using the Byte Pair Encoding (BPE) algorithm. It can be used to process text data for input into machine learning models, or to

syonfoxalmost 3 years agoJavaScript, BPE, Encoder, Decoder

npmGitHubHomepage

@syonfox/gpt-3-encoder107MIT1.4.0-rc5

Javascript BPE Encoder Decoder for GPT-2 / GPT-3. The "gpt-3-encoder" module provides functions for encoding and decoding text using the Byte Pair Encoding (BPE) algorithm. It can be used to process text data for input into machine learning models, or to

latitudegamesalmost 3 years agoJavaScript, BPE, Encoder, Decoder

ai-token-estimator966MIT1.7.1

Estimate and count tokens (incl. exact OpenAI BPE) and input costs for LLM API calls

BitsAndBytesAI13 days agollm, tokenizer, token-count, token-counter

@openfluke/welvet355Apache-2.00.3.0

TypeScript/JavaScript bindings for LOOM neural network framework with WebAssembly support - GPU-accelerated machine learning in the browser

openfluke14 days agoloom, neural-network, machine-learning, deep-learning

Packages

Lightweight debugging utility for Node.js and the browser

debug, log, debugger

Lodash modular utilities.

modules, stdlib, util

React is a JavaScript library for building user interfaces.

A querystring parser that supports nesting and arrays, with a depth limit

querystring, qs, query, url

Higher-order functions and common patterns for asynchronous code

async, callback, module, utility

babel babel-core

Babel compiler core.

6to5, babel, classes, const