Talking to an LLM — An Interactive Guide

What a model actually is

Three things to hold onto. Everything else on this page follows from them.

It is a function, not a service you connect to

You hand it text. It hands back text. That's the entire contract. There is no connection held open, no session, nothing running on your behalf between calls, and no way to reach it other than by asking it something.

People picture a thing sitting there thinking. There isn't one. Between your requests it does not exist.

Which means it also has no clock, no internet, and no way to run anything. If it appears to know today's date, you told it. If it appears to look something up, your program did the looking and pasted the answer in.

It remembers nothing, so you resend everything

A chat is an illusion you build. On turn five you are not continuing anything — you are sending turns one through four again, plus the new question, as one lump of text. The model reads the lot from cold and answers.

This is the single most expensive fact about building with these things, and almost nobody says it out loud.

A conversation therefore gets more expensive with every turn, and it gets slower too. Twenty turns in, you may be resending fifteen thousand tokens to ask a nine-word question. The fixes are all the same shape: send less, or arrange things so you aren't billed full price for the part that never changes. Prompt caching is that second one.

What people do about it

Truncate old turns, summarise them into a short note, or keep only what's relevant to the current question. All three lose something. Decide deliberately which, rather than discovering it when a user asks "what did I say earlier" and the model has no idea.

It predicts, which is why it is confident when it's wrong

Underneath, it is choosing a likely next token, then the next, then the next. Nothing in that process checks whether the result is true. Plausible and correct usually coincide, which is exactly what makes the times they don't so hard to spot.

So a made-up answer doesn't arrive hedged and nervous. It arrives in the same steady voice as a correct one, because the voice is a property of the writing, not of the knowing. If you need certainty, get it from a tool, a database or a document you retrieved — not from the model's tone.

Tokens, and why you care

Every limit you hit and every penny you spend is counted in these. Worth ten minutes.

Not words. Pieces of words.

A model doesn't read letters or words. It reads tokens — chunks that are usually about three-quarters of an English word. Common words are one token. Unusual names, long numbers, code and non-English text cost several each.

Send the invoice to Kwazulu-Natal by Friday.

Twelve words' worth of meaning, thirteen tokens — and six of them went on one place name.

A rough rule for English prose: tokens ≈ words ÷ 0.75. Don't trust it for anything else. JSON, UUIDs, base64 and minified code are all far denser than they look, which is why a payload that seems small can blow a context window.

Count before you send

There is a token-counting endpoint. Use it when you're assembling prompts from data you don't control — retrieved documents, user uploads, tool results — so you find out you're over the limit before the request fails rather than after.

The context window is a hard wall

Your prompt and the reply have to fit in the same budget. If the model can hold 200,000 tokens and your prompt is 199,000, you get a thousand tokens of answer and not a word more.

max_tokens is a cap on the reply, and it's a limit, not a target — you are charged for what's produced, not what you allowed. But set it too low and the reply stops mid-sentence with stop_reason: "max_tokens", which is a real bug people ship regularly because it only shows up on unusually long answers.

Big windows are not free

A model that accepts a million tokens will happily take them and bill you for them, on every turn they stay in the conversation. Long context is an escape hatch, not a strategy. When the same reference material is needed repeatedly, retrieval or caching costs less.

Output costs several times more than input

Reading is cheap; writing is not. On most models a token the model produces costs around five times one you send. So a chatty format — long preambles, restated questions, JSON with verbose key names — is a real line on your bill, not a style preference.

It also means the cheapest optimisation available is usually "ask for less". Telling a model to answer in one sentence rather than letting it produce four paragraphs can cut the cost of a request more than switching to a smaller model would.

The shape of a call

One request, one response, and the four fields that matter.

Everything you send

from anthropic import Anthropic

client = Anthropic()          # reads ANTHROPIC_API_KEY from the environment

msg = client.messages.create(
    model="claude-opus-5",
    max_tokens=1024,
    system="You are terse. Answer in one sentence.",
    messages=[
        {"role": "user",      "content": "Why is the sky blue?"},
        {"role": "assistant", "content": "Shorter wavelengths scatter more."},
        {"role": "user",      "content": "So why are sunsets red?"},
    ],
)

print(msg.content[0].text)

system is standing instruction — who the model is and what the rules are. messages is the transcript, alternating user and assistant, oldest first. Note that you are supplying the assistant's own previous replies: it doesn't have them.

Keep the system prompt for things that are true on every turn. Anything specific to this question belongs in the user turn. It's a real distinction, not a stylistic one — caching works on the front of the prompt, so a stable system block is one you can stop paying full price for.

Content is blocks, not a string

"content": "hello" is shorthand. Underneath, content is a list of typed blocks — text, image, tool_use, tool_result, thinking. As soon as you do anything beyond plain chat you'll be building that list yourself, so it's worth knowing the shorthand is a convenience.

Everything you get back

{
  "id": "msg_01XyZ…",
  "model": "claude-opus-5",
  "role": "assistant",
  "content": [ { "type": "text", "text": "Because by then…" } ],
  "stop_reason": "end_turn",
  "usage": { "input_tokens": 41, "output_tokens": 18 }
}

Read stop_reason every time. end_turn means it finished. max_tokens means you cut it off mid-thought. tool_use means it is asking you to go and do something. Treating all three the same is how truncated answers reach production.

usage is your ground truth for cost. Log it. Not an estimate, not a token count you made yourself — the number the provider actually billed. It's the only way to find out which feature is quietly costing you the most.

Streaming

It makes nothing faster. It changes what waiting feels like, which matters more.

Two different numbers

Time to first token is how long before anything appears. Total time is how long until it's done. Without streaming, a person experiences only the second one — a dead screen for eight seconds. With streaming they experience the first, and eight seconds of text arriving reads as fast.

Same duration. Completely different product.

There is a practical reason too: a long non-streamed request can hit a proxy or gateway timeout and die having produced everything, giving you nothing. If a call might take a while or produce a lot, stream it whether or not anyone's watching.

The tidy way to do it

The SDKs give you a context manager plus a helper for the assembled result, so you don't have to hand-reduce the event stream unless you want per-event control:

with client.messages.stream(
    model="claude-opus-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Explain sunsets."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)

final = stream.get_final_message()   # full message, usage included

Getting data back, not prose

"Respond in JSON" is a request. A schema is a guarantee. Use the guarantee.

Why asking nicely fails

Put "reply with JSON only" in a prompt and it works almost every time. Almost. Then something arrives wrapped in a code fence, or with a cheerful sentence in front of it, or with a trailing comma — and your parser throws at three in the morning on input you can't reproduce.

Constrain the output instead. Then it can't be malformed, because nothing else can be produced.

Supply the shape you want as JSON Schema and let the SDK hand you back a parsed, validated object. The failure mode moves from "unpredictable text at runtime" to "a validation error you can actually see", which is a trade worth making every time.

Shape it for the job

Give every field a description — those are read, and they improve what lands in the field. Keep required lists short so the model isn't forced to invent a value it doesn't have, and prefer a nullable field over a mandatory one it has to guess at. An enum is the strongest instruction you can give.

The knobs

There are fewer than you think, and one of them is widely misunderstood.

Temperature

How much randomness goes into picking each next token. At 0 it takes the most likely one every time. Higher, and it sometimes takes the second or third.

It is not a creativity dial. High temperature doesn't produce better ideas, it produces less predictable word choices — which is sometimes what you want and often just noise.

Extraction, classification, routing, anything you'll parse: keep it low. Copy, names, brainstorming a list of options: raise it. And note that 0 is not deterministic — batching and hardware mean the same input can still vary. Don't build a cache key on the assumption that it won't.

Thinking

Lets the model work through a problem before answering. It genuinely helps on multi-step reasoning, planning, and anything where the first instinct is often wrong.

Turn it on for hard things. Leave it off for extraction and classification, where it buys nothing and costs tokens.

On current models

Use thinking: {"type": "adaptive"} and let the model decide how much it needs. The older budget_tokens form is deprecated on 4.6-era models and rejected outright by the current ones — if you're carrying that from an old tutorial it will fail with a 400.

Prompt caching

The biggest cost lever available, and the one most people never touch.

Stop paying full price for the part that never changes

Most real prompts have a large unchanging front — a system prompt, tool definitions, a style guide, a document being discussed — and a small changing tail. Without caching you pay full price for that front on every single turn.

Mark where the stable part ends and reads of it become dramatically cheaper. On a long system prompt over a twenty-turn conversation this is not a rounding error; it's most of the bill.

One rule makes it work: stable content first, volatile content last. The cache matches on an exact prefix, so a timestamp, a session id or a shuffled list near the top invalidates everything after it. If you've ever cached and seen no saving, that is almost certainly why.

Reading the numbers

Writing to the cache costs slightly more than a normal read, so it pays back from the second hit onward — not worth it for genuinely one-shot calls. Watch cache_creation_input_tokens against cache_read_input_tokens in usage. If you're mostly seeing creations, your prefix isn't stable and you're paying the premium without the benefit.

When it fails

It will. These are not exceptional conditions; they're Tuesday.

429 is normal. Handle it properly.

Rate limits are a fact of the service, not a sign you've done something wrong. Retry with a wait that grows each time — and add a small random amount to it, or every one of your workers will retry in perfect unison and rebuild the spike that caused the problem.

Distinguish the classes before retrying. 429 and 529 are worth another go. 500 sometimes. 400 never — your request is malformed and it will be malformed the second time too. Retrying a 400 in a loop is how a bug becomes an outage.

The rest of the checklist

Set a timeout, or a hung request holds a worker forever. Cap total attempts. Consider whether a retry is safe at all — if the call had a side effect through a tool, doing it twice may not be harmless. And when you finally give up, fail with something a person can act on rather than a spinner that never stops.

The failures that don't raise

A 200 response is not the same as a good response. It can come back truncated (stop_reason: "max_tokens"), refused, empty, or confidently wrong. None of those throw an exception, so none of them will show up in your error rate.

Which is why the interesting thing to monitor is not exceptions. Log the assembled prompt, the stop_reason, the usage and the reply for a sample of real traffic. Almost every "the AI got worse" report is answered by reading twenty of those, and answerable by nothing else.

Build a request

Everything above, on one screen. Change the settings on the left and watch the payload, the token count and the price move with them. Nothing here is sent anywhere — it's a model of the call, not the call.

Try this. Drag the conversation up to twenty turns and watch the cost per call climb without you asking a longer question — that's the resending. Now tick Cache the stable prefix and watch what happens to the same conversation. Then untick it and turn on tools instead, and notice that tool definitions are part of the prompt too, on every single turn.

What actually goes wrong

The ones that cost an afternoon, in rough order of how often they happen.

Shipping the key to the browser

Calling the API from front-end JavaScript puts your key in the page. Anyone can read it and spend your money.

Calls go from your server. The browser talks to your endpoint, your endpoint talks to the model. That also gives you the one place to put auth, rate limiting per user, and logging — all of which you will want anyway.

Never reading stop_reason

The reply looked fine in testing. In production a long answer hits max_tokens, stops mid-sentence, and is stored as though it were complete.

Branch on it. Truncated means retry with a higher cap or ask for less. tool_use means you have work to do before the turn is over.

A volatile prefix that kills the cache

Caching is on, savings are zero, and nobody can see why. Something at the top of the prompt changes every call — a timestamp, a session id, the user's name.

Move it below the cache breakpoint. Compare cache_creation_input_tokens with cache_read_input_tokens to confirm you're getting hits rather than paying the write premium every time.

Retrying everything, immediately, forever

A tight retry loop with no backoff turns a brief rate limit into a sustained one, and a malformed request into an infinite one.

Grow the wait, add jitter, cap the attempts, and never retry a 400.

Prompt tuning with no way to tell if it helped

Someone rewrites the system prompt, tries three examples by hand, and ships. Nobody finds out what it broke until a customer does.

Twenty saved examples with known-good answers, run on every prompt change, will tell you more than a week of intuition. It is an afternoon's work and the highest-leverage thing on this page.