Real-time LLM Inference on Standard GPUs: 3k tokens/s per request

Posted 5 hours ago by NicoConstant
106 points

https://blog.kog.ai/real-time-llm-inference-on-standard-gpus-3-000-tokens-s-per-request/

23 comments

Loading..

Loading..

Loading..

Loading..

Loading..

Loading..

Loading..

Loading..

Loading..

Loading..

Loading..

Loading..

Loading..

Loading..

Loading..

Loading..

Loading..

Loading..

Loading..

Loading..

Loading..

Loading..

Loading..