Model deployment and inference
Running models on your own infrastructure when data residency, unit cost or latency make a hosted API the wrong answer.
The problem
Self hosting is cheaper per token and dearer per engineer hour. Whether it is right depends on volume and constraints, and most people are told to do it before anyone has done the arithmetic.
What the work involves
- An honest build versus buy calculation before anything is provisioned
- GPU sizing against your real concurrency, not a benchmark
- Quantisation and batching, with the quality cost measured rather than assumed
- Autoscaling that accounts for cold start on large weights
- Observability: latency percentiles, queue depth, utilisation, cost per request
Typical stack
vLLM
Ollama
Docker
Kubernetes
Terraform
AWS
Hetzner
Indicative rather than fixed. The right choice depends on what you already run, and inheriting a stack your team knows usually beats introducing a better one nobody can maintain.
Got a deployment problem?
Technical detail welcome. The more concrete the problem, the more useful the first reply.