11 — Technology category · TR-110.50
Model Serving & Inference
Running trained models behind an endpoint with scaling, batching and monitoring of what they predict.
- Inference Serving
- Model Deployment
In scope
- Online and batch inference
- GPU scheduling
- Prediction monitoring
Example technologies · 4
Well-known technologies in this category, named as facts. Not a shortlist or an endorsement.
- KServe
- vLLM
- NVIDIA Triton Inference Server
- BentoML