Inference
Inference is the runtime phase in which a trained model produces outputs from new inputs, distinct from the earlier training phase where weights are learned. For language models this means generating tokens one at a time conditioned on the prompt and prior outputs. Inference cost depends on model size, context length, and decoding settings, and is what end users pay for per request.
Definition
Inference is the runtime phase in which a trained model produces outputs from new inputs, distinct from the earlier training phase where weights are learned. For language models this means generating tokens one at a time conditioned on the prompt and prior outputs. Inference cost depends on model size, context length, and decoding settings, and is what end users pay for per request.
Examples
- Streaming tokens back to a chat UI as the model generates a response.
- Batching 50 classification requests together to reduce per-call GPU overhead.
- Running a smaller distilled model on-device to keep latency under 200 ms.
Related terms
No credit card required
Frequently asked
- What is Inference?
- Inference is the runtime phase in which a trained model produces outputs from new inputs, distinct from the earlier training phase where weights are learned. For language models this means generating tokens one at a time conditioned on the prompt and prior outputs. Inference cost depends on model size, context length, and decoding settings, and is what end users pay for per request.
- How is Inference used in practice?
- Inference is commonly used in scenarios such as: Streaming tokens back to a chat UI as the model generates a response.; Batching 50 classification requests together to reduce per-call GPU overhead.; Running a smaller distilled model on-device to keep latency under 200 ms..
- How does yno.ai support Inference?
- yno.ai supports Inference as part of its agentic AI platform — combining multiple frontier models with extensible tool use.
- What other concepts are related to Inference?
- Inference relates to Large Language Model, Temperature (Sampling), Top-p / Top-k Sampling, Fine-Tuning. Each links to its own glossary page below.