gpt-encoder

GPT-2 and GPT-3 use byte pair encoding to turn text into a series of integers to feed into the model. latitudegames implemented this in Node.js. This is an implementation of their encoder for use in the browser, re-written in Typescript.

Installation

npm install @beskar-labs/gpt-encoder

Usage

import { encode } from '@beskar-labs/gpt-encoder';

const tokens = encode(
  'This is an example sentence to try encoding out on!'
).length;