Efficiently Serving Open Source LLMs
Published in
5 min readAug 14, 2023
This article explains my personal experiences using 6 common methods for serving open source LLMs: AWS Sage Maker, Hugging Face, Together.AI, VLLM and Petals.ml.
The struggle…
You’ve felt the pain, struggle and glory of serving your own fine-tuned open source LLM, however, you ultimately decided to return to Open AI or Anthropic due to cost, inference time…
