When using large language models, we typically have two perceptions of “speed”: one is how long it takes for the first character to appear after a question is posed; the other is whether the text generated after the response begins flows smoothly.
This is very similar to having a conversation with a person: after you ask a question, the other person might pause for a moment before speaking. TTFT (Time to First Token) measures this “wait time before speaking.” We can explain this clearly from three perspectives.
I. What Is TTFT (Time to First Token)?
Large language models do not generate answers all at once. They first use a tokenizer to convert text into tokens, then generate output one by one. A token is not necessarily equivalent to a single character: it could be a Chinese character, part of an English word, or even include punctuation or multiple characters.
Different models may use different tokenizers, so the same sentence may not be split into the same number of tokens. Therefore, a token is the basic unit for internal text processing within the model, not a fixed unit of “word count.”
TTFT refers to the time elapsed from the start of the request until the first output token becomes available. When measured from the client’s perspective, it can typically be approximated as:
Client TTFT ≈ Time when the first non-empty generated content is received − Time when the request was sent
The reason this is referred to as an “approximation” is that the client receives streamed events, which do not necessarily correspond one-to-one with the model’s internal tokens. Some initial events contain only roles or metadata, with no content visible to the user.
TTFT is also not equivalent to the model’s overall speed. Several common metrics can be summarized as follows:
- TTFT: how long it takes for the model to start responding;
- Subsequent generation speed (often described using ITL or TPOT): how smoothly text appears after the model starts responding;
- Total delay: how long it takes to generate the complete answer.
A model may produce the first character very quickly but generate subsequent text slowly. Therefore, TTFT only answers “how long it takes to start,” not “how long it takes to finish,” and certainly does not measure whether the answer is correct.
For example, Model A might start responding after half a second but take a long time to finish; Model B might not start until two seconds later, but generate text more quickly afterward. For short replies, Model A might offer a better user experience; for long articles, Model B might actually finish sooner.
II. What components make up TTFT?
From a client-to-port perspective, before the first token appears, the request must travel through a complete chain. TTFT is not the latency generated by any single step, but rather the cumulative result of these waiting times:
Uplink request → Request preprocessing and queuing → Prefill → Downlink of the first token
This is a simplification for clarity. Different services may perform authentication, tokenization, queuing, and scheduling in varying orders, and the timing start points may differ across benchmarking tools.
- Network Latency: The request must travel from the client to the server, and the first valid output must then be transmitted back to the client. Network distance, congestion, gateways, and proxies can all introduce additional wait times.
- Request Preprocessing: The server must parse the request and use a tokenizer to convert the prompt into tokens that the model can process; the specific duration depends on the service implementation and the input.
- Queueing Time: When a large number of requests arrive simultaneously, new requests may have to wait for computational resources or service scheduling. In this case, TTFT will increase, though this does not necessarily mean the model itself has slowed down.
- Prefill Phase: The model processes all input tokens in parallel, establishes the intermediate states required for subsequent generation (typically including a KV cache), and computes the first output token. This phase is typically computationally intensive; the longer the input, the greater the workload.
The “input” here is not limited to the single sentence a user has just entered. In chat systems, knowledge base Q&A, or agents, a seemingly simple question may be backed by a long history of conversations, retrieved documents, and tool instructions. Just because the question visible to the user is short does not mean the context the model actually needs to process is short as well.
Only after the first token is generated does the model enter the Decode phase, where it generates subsequent tokens one by one based on the existing context. This marks the end of TTFT; any subsequent speed variations relate to generation speed. This is also the most fundamental difference between TTFT and “how quickly the entire answer is generated.”
However, long inputs aren’t the only factor. Model size and architecture, hardware, the inference engine, cold starts, cache hits, and network latency can all affect the results. For inference models, additional computations required before the final answer is produced may also increase the wait time for the first visible content.
Therefore, the TTFT measured by the client represents the result of the entire service chain and cannot be directly equated with the computational time taken by the GPU or the model.
III. How to Reduce TTFT?
The key to reducing TTFT is to minimize unnecessary waiting time before the first token appears. The previous section broke down TTFT into several components, and optimization can be carried out along the same chain.
- Reduce unnecessary input: Excessively long system prompts, constantly accumulating conversation history, and retrieved information that is not closely related to the question all increase the workload of the prefill process. Summarizing old conversations while retaining only relevant content—without losing key information—is typically more effective than blindly expanding the context.
- Utilize the Prompt Cache:If multiple requests share the same prompt prefix—such as fixed system prompts, tool definitions, or public data—repeated input processing can be reduced when the service supports it and a cache hit occurs.
- Reducing Queuing and Invalid Scheduling:When concurrent requests exceed system capacity, TTFT will increase due to resource waiting, even if individual model computations are fast. More reasonable request scheduling and batch processing can strike a balance between response speed and system throughput.
- Choose the Right Model for the Task:Larger models or inference models requiring additional computation are not necessarily suitable for all tasks. For simple, highly interactive problems, smaller or faster models may provide a better user experience; for complex inference, however, users may need to accept longer wait times.
Streaming output is also important, but it’s crucial to understand what it optimizes. It allows content that has already been generated to be displayed to the user as early as possible, rather than waiting until the complete answer is generated and then appearing all at once; however, it does not reduce the computation time required for the model to produce the first token. Compared to non-streaming returns, it primarily makes generated content visible to the user sooner.
Finally, a lower TTFT does not necessarily mean a better model. It only measures how quickly output begins; it does not measure whether the answer is correct. The true user experience must also take into account the speed of subsequent generation, the time required for a complete answer, and the quality of the answer.
TTFT ultimately answers a very simple question: After I send a request, how long do I have to wait before I see the system actually start responding?
