text
What are your favorite text chunkers/splitters? 
What are your favorite text chunkers/splitters? 
new_line
text
Mine are: 
Mine are: 
new_line
text
- 
- 
link
https://github.com/benbrandt/text-splitter
https://github.com/benbrandt/text-splitter
text
 (Rust/Python, battle-tested, Wasm version coming soon) 
 (Rust/Python, battle-tested, Wasm version coming soon) 
new_line
text
- 
- 
link
https://github.com/umarbutler/semchunk
https://github.com/umarbutler/semchunk
text
 (Python, really performant but some issues with huge docs)
 (Python, really performant but some issues with huge docs)
new_line
new_line
text
I tried the huge Jina AI regex, but it failed for my (admittedly messy) documents, e.g. from EUR-LEX. Their free segmenter API is really cool but unfortunately times out on my huge docs (~100 pages): 
I tried the huge Jina AI regex, but it failed for my (admittedly messy) documents, e.g. from EUR-LEX. Their free segmenter API is really cool but unfortunately times out on my huge docs (~100 pages): 
link
https://jina.ai/segmenter/
https://jina.ai/segmenter/
new_line
new_line
text
Also, I tried to write a Vanilla JS chunker with a simple, adjustable hierarchical logic (inspired from the above). I think it does a decent job for the few lines of code: 
Also, I tried to write a Vanilla JS chunker with a simple, adjustable hierarchical logic (inspired from the above). I think it does a decent job for the few lines of code: 
link
https://do-me.github.io/js-text-chunker/
https://do-me.github.io/js-text-chunker/
new_line
new_line
text
Happy to hear your thoughts! 
Happy to hear your thoughts! 
What are your favorite text chunkers/splitters? 
Mine are: 
- https://github.com/benbrandt/text-splitter (Rust/Python, battle-tested, Wasm version coming soon) 
- https://github.com/umarbutler/semchunk (Python, really performant but some issues with huge docs)

I tried the huge Jina AI regex, but it failed for my (admittedly messy) documents, e.g. from EUR-LEX. Their free segmenter API is really cool but unfortunately times out on my huge docs (~100 pages): https://jina.ai/segmenter/

Also, I tried to write a Vanilla JS chunker with a simple, adjustable hierarchical logic (inspired from the above). I think it does a decent job for the few lines of code: https://do-me.github.io/js-text-chunker/

Happy to hear your thoughts! 
👀
cschroeder John6666
❤️
Felladrin
/avatars/f32291df2054c1bb4a01889d1b41c0d5.svg
Christopher Schröder
cschroeder
user