GigaToken: ~1000x faster Language model tokenization

Posted 2 hours ago by syrusakbary
163 points

https://github.com/marcelroed/gigatoken/

9 comments

Loading..

Loading..

Loading..

Loading..

Loading..

Loading..

Loading..

Loading..

Loading..