What exactly is a context window?

The maximum number of input tokens a model can read in a single call — system prompt, conversation history, tool definitions, retrieved documents, and the latest user message all count against it. It's separate from the output cap, which limits how many tokens the model can generate in its reply.

What happens if my prompt exceeds the context window?

Depends on the provider. Most APIs reject the request outright with an error naming the token count and the limit. Some client libraries or chat interfaces instead silently truncate the oldest turns of a conversation to make room, which can quietly drop instructions or context the model needed — often worse than an error, because nothing tells you it happened.

Does using less than the full window guarantee good output?

No. Fitting inside the window only means the model can technically read everything — it doesn't mean the model attends to all of it equally well. Research on long-context models documents a 'lost in the middle' effect: information placed in the middle of a long prompt is recalled less reliably than information at the start or end, even well under the token limit.

How do I count tokens before sending a request?

Roughly, English prose runs about 4 characters per token and dense code closer to 3.5, which is enough for sizing decisions. For an exact count, use the model vendor's own tokenizer (OpenAI's tiktoken, Anthropic's count-tokens endpoint, Google's count_tokens) — different vendors use different tokenizers, so the same text produces different exact counts across models.

Guide

Context Window Fit: Why Long Prompts Fail

A context window is a hard token budget, not a soft suggestion — and the failure mode when you exceed it isn't always an error.

By Buğra SözeriPublished September 25, 2026

Every hosted large language model has a fixed cap on how many tokens it can read in one call: the context window. It sounds like a simple limit — stay under it and you’re fine — but two things about it trip up almost everyone building on top of an API: what actually counts toward the number, and what happens once you cross it. You can check exactly how a given prompt lands against every major model’s window with the context window visualizer.

What counts against the window

Everything the model reads before it starts writing counts: the system prompt, every prior turn in a multi-turn conversation, tool and function-call schemas passed to the model, retrieved documents in a RAG pipeline, and the latest user message. None of that is optional or free — a long system prompt repeated on every call in a chat app is a common, easy-to-miss source of overflow, because it’s invisible in the UI but resent in full on every single request.

Output is billed and limited separately, with its own cap that is usually far smaller than the context window itself. A model advertising a very large context window can still only emit a modest number of tokens per reply — the two numbers answer different questions and shouldn’t be confused.

What actually happens on overflow

The failure mode isn’t uniform across providers, which is part of why it’s easy to get wrong in production:

Hard rejection.Most direct API calls that exceed the documented token limit return an explicit error naming the request’s token count against the model’s limit. This is the safest failure mode because it’s loud and immediate.
Silent truncation.Some client wrappers and chat interfaces instead drop the oldest messages from a conversation history to make the next call fit, without surfacing that anything was removed. The model then responds as if it never saw the dropped context — which can look like the model “forgetting” instructions or facts that were, in fact, silently cut.

Neither failure mode is something you want to discover in production. Sizing the prompt against the window before sending it — or building the check into your pipeline — is the more reliable habit, and it’s exactly what the context window visualizer does for a pasted prompt: it shows the fill percentage per model and flags red once you’re past roughly 80% utilization.

Fitting isn't the same as being read well

A prompt that fits comfortably inside the window can still produce worse answers than a shorter one. Independent long-context research has documented a “lost in the middle” effect: models recall information placed near the start or end of a long prompt more reliably than information buried in the middle, even when the whole prompt is well under the token limit. This isn’t a bug in any one vendor’s model — it shows up across architectures and is a property of how attention degrades over very long sequences.

The practical implication: treat “does it fit” and “will the model actually use it well” as two separate questions. For retrieval-heavy workloads, a smaller set of well-chosen chunks placed near the edges of the prompt often outperforms a maximal dump that technically fits but buries the important passage in the middle.

Estimating tokens without a full tokenizer

For quick sizing, roughly 4 characters per token for English prose and closer to 3.5 for dense code is close enough to plan around — that heuristic is what the token counterand the visualizer both use. For a number you can rely on before shipping — or before signing off on a cost estimate — run the same text through the vendor’s own tokenizer, since different vendors use different tokenization schemes and the exact count will differ between them even for identical text. Once you know the token count, the LLM cost calculator turns it into a per-call price.

Frequently asked questions

What exactly is a context window?: The maximum number of input tokens a model can read in a single call — system prompt, conversation history, tool definitions, retrieved documents, and the latest user message all count against it. It's separate from the output cap, which limits how many tokens the model can generate in its reply.
What happens if my prompt exceeds the context window?: Depends on the provider. Most APIs reject the request outright with an error naming the token count and the limit. Some client libraries or chat interfaces instead silently truncate the oldest turns of a conversation to make room, which can quietly drop instructions or context the model needed — often worse than an error, because nothing tells you it happened.
Does using less than the full window guarantee good output?: No. Fitting inside the window only means the model can technically read everything — it doesn't mean the model attends to all of it equally well. Research on long-context models documents a 'lost in the middle' effect: information placed in the middle of a long prompt is recalled less reliably than information at the start or end, even well under the token limit.
How do I count tokens before sending a request?: Roughly, English prose runs about 4 characters per token and dense code closer to 3.5, which is enough for sizing decisions. For an exact count, use the model vendor's own tokenizer (OpenAI's tiktoken, Anthropic's count-tokens endpoint, Google's count_tokens) — different vendors use different tokenizers, so the same text produces different exact counts across models.

Sources & references

Authoritative references cited by this piece. Verified by Buğra Sözeri on the dates shown and re-checked at every deploy.

OpenAI — Models reference — Published context window and max-output token limits per model(as of 2026-09-25)
Anthropic — Models overview — Context window and output limits for Claude models(as of 2026-09-25)
Liu et al. — Lost in the Middle: How Language Models Use Long Contexts — Peer-reviewed source for the degraded-recall-in-the-middle effect in long-context prompts(as of 2026-09-25)