Token Text Splitter

Splits a raw text string by first converting the text into BPE tokens, then split these tokens into chunks and convert the tokens within a single chunk back into text.

This section is a work in progress. We appreciate any help you can provide in completing this section. Please check our Contribution Guide to get started.

Last updated