Token Text Splitter
Splits a raw text string by first converting the text into BPE tokens, then split these tokens into chunks and convert the tokens within a single chunk back into text.
![](https://docs.flowiseai.com/~gitbook/image?url=https%3A%2F%2F1820151947-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252Fy8ifwt9BYklr92KDdr48%252Fuploads%252Fgit-blob-79be6075832da1644a671f5b9a885c70360839ff%252Fimage%2520%28156%29.png%3Falt%3Dmedia&width=768&dpr=4&quality=100&sign=2f6c3217&sv=1)
This section is a work in progress. We appreciate any help you can provide in completing this section. Please check our Contribution Guide to get started.
Last updated