When a model serves several apps or a group of people, a single-machine script stops coping. vLLM is an open-source inference and serving framework on GitHub that batches requests and exposes service endpoints in common formats, aimed at production deployment. It needs your own compute, and supported architectures and hardware change by version, so check the repository docs; configure auth and rate limits before going live.
What it does
- Batch scheduling raises throughput for concurrent requests
- Serves inference endpoints in common formats
- Supports many mainstream open architectures
- Mechanisms for long context and serving
- Usually deployed alongside containers
Who it is for
- Serving a model to several internal apps
- Assessing throughput and queuing as concurrency grows
- Serving long context or structured output
- Offering a model API from your own servers
How to get started
- Run it on a test machine and load it with realistic concurrency
- Set batching and concurrency from observed data
- Configure auth and rate limits; never expose a bare API
