Important: This documentation covers Yarn 1 (Classic).
For Yarn 2+ docs and migration guide, see yarnpkg.com.

Package detail

nai-js-tokenizer

NovelAI424MIT1.1.0TypeScript support: included

BPE tokenizer used for the NovelAI frontend.

gpt, gpt2, pile, sentencepiece, nerdstash, tokenizer, novelai

readme

The BPE tokenizer used for NovelAI frontend.