# Using an LLM API in Your App > Calling a hosted language model is a normal HTTP request: you POST a list of messages, you get back generated text - and this guide builds the mental model, the cost picture, and the reliability habits you need to ship it without a foot-gun. --- # Using an LLM API in Your App You've used a chat assistant in a browser, and now you want one *inside* your own app - to summarize, to answer, to draft. Somewhere along the way the phrase "call the LLM API" showed up, and it sounds like it needs a research lab and a GPU farm. It doesn't. Here's the part nobody says out loud: a hosted language model is reached the exact same way as any other web service. You send an HTTP request, you get a response back. If you've ever called a weather API or a payments API, you already know 90% of this. The model is the unusual part; the *calling* is ordinary. This guide installs that mental model first, then walks you through what actually costs money, and finally the habits that keep a real feature from embarrassing you in production. > ⏭️ New to the idea of an API at all? Read [What an API Actually Is](/guides/what-an-api-is) first - this guide assumes you're comfortable with the idea of one program asking another for something over HTTP. ## How to read this - **Want it to finally make sense?** Read in order. We start with the request/response shape (it really is just an API call), then cover tokens and cost so the bill never surprises you, then the reliability habits that separate a demo from a feature. - **Already calling the model and hitting walls?** Jump to [Phase 3: Building Reliably](03-building-reliably.md) - non-determinism, hallucinations, timeouts, retries, and asking for structured output. ## The phases 1. **[It's Just an API Call](01-its-just-an-api-call.md)** - an LLM API is a normal HTTP request. You POST a list of messages (system, user), you get back generated text. The annotated request and response, provider-neutral. 2. **[Tokens, Context & Cost](02-tokens-context-and-cost.md)** - what a token is, the context window (the model's limited short-term memory), why you pay per token, and why long conversation histories cost more and can overflow. Plus streaming for responsiveness. 3. **[Building Reliably](03-building-reliably.md)** - the model is non-deterministic, it can be confidently wrong, it can be slow, and it can fail. How to handle errors, timeouts, and retries; how to ask for structured output; and how not to ship a foot-gun. > This guide deliberately stops at *how to call the thing well*. Getting the model to actually do what you want - writing the instructions - is its own craft, covered in [Prompt Engineering, Plainly](/guides/prompt-engineering-plainly). --- # It's Just an API Call If you've been picturing some special "AI connection" - a socket, a model loaded into your process, a thing you have to install - let that picture go. Calling a hosted model is an ordinary HTTP request to a URL, with a JSON body and an API key in the header. The same shape as nearly every web API you've ever touched. What's different is only *what you send* and *what comes back*. Let's look at exactly that. ## The mental model: you send a conversation, you get back the next line **What it actually is.** A chat-style LLM API takes a *conversation so far* and returns *what the assistant would say next*. You hand it a list of messages - who said what - and it generates one more message: the assistant's reply. That's the whole interaction. There's no hidden session living on the server remembering your last call. Each request is self-contained: the model only knows what's in the messages you send *this time*. (That detail matters a lot in Phase 2, so tuck it away.) ```mermaid sequenceDiagram participant App as Your app participant API as The model API App->>API: HTTP POST messages
[system] "You are helpful."
[user] "What's a closure?" Note over API: reads the whole list,
generates the next message API-->>App: response
[assistant] "A closure is a function that ..." ``` 📝 **Terminology.** This style is usually called the **chat completions** API (you give it a chat, it *completes* it with the next turn). Different providers name their endpoint slightly differently, but the shape - a list of role-tagged messages in, one assistant message out - is the same across the major ones. ## The three roles Each message in the list has a **role**. Three you'll use constantly: - **`system`** - your standing instructions to the model. The personality, the rules, the job. "You are a terse assistant that answers in one sentence." The user usually never sees this. It's how you set the model's behavior for the whole conversation. - **`user`** - what the person typed. The question, the request, the text to summarize. - **`assistant`** - what the model said *previously*. You only include these when you're continuing a multi-turn conversation, so the model can see what it already told the user. For a single one-shot request, you send a `system` message and a `user` message, and that's enough. ## A real request Here's a complete, provider-neutral request. It's `curl`, but it's the same JSON whatever language you're in: ```console $ curl https://api.example-llm.com/v1/chat/completions \ -H "Authorization: Bearer $LLM_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "some-chat-model", "messages": [ { "role": "system", "content": "You are a concise assistant. Answer in one short paragraph." }, { "role": "user", "content": "In plain terms, what is a closure in programming?" } ] }' ``` *What just happened:* You made a `POST` to the model's endpoint. Three things do the work: - **The header `Authorization: Bearer ...`** carries your API key, so the provider knows it's you (and whom to bill). We read it from an environment variable, `$LLM_API_KEY` - never paste the key itself into the command or your code (more on that below). - **`"model"`** picks *which* model answers. Providers offer several, trading speed and cost against capability. - **`"messages"`** is the conversation - here, a `system` instruction setting the behavior, then the `user` question. Notice what's *not* here: no GPU, no model download, no special protocol. A URL, a key, some JSON. ## A real response The reply is JSON too. Trimmed to the parts that matter: ```json { "id": "resp_8c1f2a", "model": "some-chat-model", "choices": [ { "index": 0, "message": { "role": "assistant", "content": "A closure is a function that remembers the variables from the place where it was created, even after that place has finished running. It carries that little bundle of context around with it, so it can still use those variables later." }, "finish_reason": "stop" } ], "usage": { "prompt_tokens": 31, "completion_tokens": 52, "total_tokens": 83 } } ``` *What just happened:* The model generated the next message and handed it back. The pieces you'll actually use: - **`choices[0].message.content`** is the generated text - the answer. This is the string you show your user. (It's an array because some APIs can return more than one candidate reply, but you'll almost always read `choices[0]`.) - **`finish_reason: "stop"`** means the model finished naturally. If you ever see `"length"` instead, it means the reply got cut off because it hit a length limit - a sign to allow more output room (Phase 2 explains the limit it hit). - **`usage`** counts the tokens this call used: input (`prompt_tokens`), output (`completion_tokens`), and the total. That's what you're billed on, and it's why Phase 2 exists. For now, just notice the model *tells you* the count on every call. So the entire loop is: build a `messages` list → POST it → read `choices[0].message.content`. Everything else in this guide is about doing that *well*. ## Continuing a conversation Because the server doesn't remember anything between calls, "memory" is something *you* provide - by sending the earlier turns back each time. To ask a follow-up, you append the model's last reply and the new user message to the list: ```json { "model": "some-chat-model", "messages": [ { "role": "system", "content": "You are a concise assistant." }, { "role": "user", "content": "What is a closure?" }, { "role": "assistant", "content": "A closure is a function that remembers the variables ..." }, { "role": "user", "content": "Can you give me a tiny example?" } ] } ``` *What just happened:* You replayed the whole conversation and added the new question at the end. The model reads the entire list as context and answers the follow-up knowing what it said before. This is the idea that surprises people most: **the conversation is something you carry and resend, not something the server holds for you.** It's also exactly why long chats get expensive - you'll feel that in Phase 2. ## Keep your API key out of your code ⚠️ **Gotcha - the big one.** That API key is a password that spends your money. Do **not** write it into your source code, and *never* commit it to git. People do this constantly, push to a public repo, and wake up to a key that strangers are running up a bill on. Scanners crawl public repositories looking for exactly these strings. The fix is the same as for any secret: load it from an environment variable or a secrets manager at runtime, and keep the actual value out of the files you check in. There's a whole guide on doing this properly, read it before you ship anything: [Secrets Management](/guides/secrets-management). A second, related trap: don't call the model directly from your *frontend* (browser or mobile) code, because anything shipped to the user's device can be read by the user. The key belongs on a server you control, which calls the model on the user's behalf. ## Recap 1. An LLM API is a normal **HTTP POST** - a URL, your key in a header, JSON in the body. 2. You send a **`messages`** list of role-tagged turns; the main roles are **`system`** (your instructions), **`user`** (the person), and **`assistant`** (the model's earlier replies). 3. You read the answer from **`choices[0].message.content`**, and `usage` tells you the token cost of the call. 4. The server is **stateless** - to continue a conversation, you resend the prior turns yourself. 5. The API **key is a password.** Keep it out of code and out of the frontend; load it from a secret at runtime. Next: what those tokens in the `usage` block actually are, why they're the unit of both memory and money, and how to keep both under control. --- # Tokens, Context & Cost In Phase 1 you saw a `usage` block counting `prompt_tokens` and `completion_tokens`. That word - *token* - is the unit the whole economy of LLMs runs on: how much the model can "hold in its head," and what you're billed in. Get a feel for tokens and two mysteries dissolve at once: why your bill is what it is, and why long conversations start failing. This phase saves you from a surprise invoice and a confusing "the model forgot the start of our chat" bug. ## What a token is **What it actually is.** A **token** is a chunk of text - usually a short piece of a word, sometimes a whole short word, sometimes punctuation. Models don't read letter by letter or word by word; they read in chunks. "Cat" might be one token; "unbelievable" might be three (`un`, `believ`, `able`); a space and the next word often travel together. **Why people get this wrong.** The common assumption is "one token = one word." Close, but it'll mislead you. A rough working rule for English is that a token is a bit less than a word, so a paragraph is more tokens than it has words. Don't treat that as exact - providers publish tools to count tokens precisely, and the `usage` field tells you the real count after each call. Use those for anything that matters; the rough rule is only for back-of-envelope estimates. 📝 **Terminology.** "Tokenize" just means *chop the text into these chunks*. Everything the model reads (your prompt) and everything it writes (its reply) is measured in tokens. ## The context window - the model's short-term memory **What it actually is.** The **context window** is the maximum number of tokens a model can consider at once - its entire short-term memory for a single request. Crucially, **the input and the output share this same budget.** Everything you send (system message, the whole conversation history, the user's question) *plus* everything the model generates back has to fit inside that one window. ```text ┌──────────────── the context window (a fixed token budget) ────────────────┐ │ │ │ [system] [earlier turns ...........] [your new question] [the reply] │ │ └──────────────── what you send (input) ──────────────┘ └── output ──┘ │ │ │ └────────────────────────────────────────────────────────────────────────┘ if the whole thing won't fit, something has to give ``` **What it does in real life.** The window is large on modern models - easily enough for a long document or a substantial conversation - but it's *finite*. As your input grows (long document, long chat history), you leave less room for the output, and eventually risk not fitting at all. ⚠️ **Gotcha.** Two failure modes come straight from this shared budget: - **The reply gets cut off.** If your input eats most of the window, little room is left for the answer, and the model stops mid-sentence. Remember `finish_reason: "length"` from Phase 1? That's this. Fix: leave headroom - shorter input, or explicitly reserve more space for output. - **The request is rejected.** If the input alone exceeds the window, the API errors before generating anything. You can't just keep appending to a conversation forever. **Why this saves you later.** This is the real reason a long-running chat eventually breaks or "forgets" the beginning. The cure is to keep only what matters in the window - trim or summarize old turns rather than blindly resending the entire history (which, from Phase 1, *you* are the one resending). ## You pay per token **What it actually is.** With a hosted model, you're billed by tokens used - both sent and generated. There's no flat per-request price; a call that processes a long document and writes a long answer costs more than a one-line question with a one-line reply. A few things worth knowing, stated plainly rather than with invented numbers: - **Input and output are often priced differently.** Output tokens are commonly billed at a higher rate than input tokens. Exact rates vary by provider and model - **check current pricing on the provider's site**, because it changes. - **Bigger, more capable models cost more per token** than smaller, faster ones. Part of building well is using a cheaper model where it's good enough and saving the expensive one for hard requests. - **The `usage` block is your meter.** Every response tells you exactly how many tokens that call cost. Log it - that's how you find the request type quietly draining your budget. 💡 **Key point.** Two levers control your bill: *tokens sent* and *tokens generated*. Control the first by trimming history and not stuffing in irrelevant context; control the second by capping output length where a short answer will do. ## Long histories cost more - and can overflow These two ideas - the window and the per-token price - collide in the most common real-world bug. Because you resend the whole conversation on every turn (Phase 1), each new message makes the *next* request bigger: ```text turn 1: send [system + Q1] → small, cheap turn 2: send [system + Q1 + A1 + Q2] → bigger turn 3: send [system + Q1 + A1 + Q2 + A2 + Q3] → bigger still ... turn N: send the entire conversation, every time → slow, costly, and eventually too big ``` ⚠️ **Gotcha.** A naive chat feature that just keeps appending turns gets *more expensive and slower with every message*, and one day a long conversation crosses the window limit and starts erroring. This catches people in production after the demo worked fine with three short messages. Plan for it: cap the history you resend, drop or summarize old turns, keep the system message lean. ## Streaming - for when waiting feels broken **What it actually is.** Normally you wait for the model to finish, then get the whole reply at once. **Streaming** instead sends the reply to you token by token as it's generated, so you can show text appearing live - the typewriter effect you've seen in chat apps. **What it does in real life.** Streaming doesn't make the model faster or cheaper - total tokens and total time are about the same. What it changes is *perceived* speed: the user sees words within a moment instead of staring at a spinner for several seconds. For anything interactive, that difference is the whole experience. **The trade-off.** Streaming is a bit more work to handle: instead of one JSON response, you read a sequence of small chunks and stitch them together. For a background job, don't bother - wait for the full response. For a user watching a chat box, it's usually worth it. ## Recap 1. A **token** is a chunk of text, a bit smaller than a word; both your input and the model's output are measured in tokens. 2. The **context window** is a fixed token budget that input *and* output share - overflow it and replies get cut off or the request is rejected. 3. You **pay per token**, input and output (often at different rates, bigger models cost more) - check current pricing, and watch the `usage` meter. 4. Because you resend the whole history each turn, **long conversations get costlier, slower, and eventually too big** - trim or summarize. 5. **Streaming** shows the reply as it's generated: same cost and total time, far better perceived responsiveness for interactive use. Next: the model will sometimes be wrong, slow, or unavailable - and it won't warn you. Phase 3 is the set of habits that turn a fragile demo into a feature you can stand behind. --- Type anything and watch it split into tokens - the unit you actually pay for: ```playground-tokens ``` Watch it animated: [tokens and context windows](/explainers/TokensContext.dc.html) --- # Building Reliably The first time you wire a model into your app and it answers your question, it feels like magic. The second time, you ask the *same* question and get a slightly different answer, and the magic curdles into worry. Then it confidently states something false, or takes eight seconds, or returns an error mid-demo - and you realize you're not calling a database that returns the same row every time. You're calling something that *generates*. That's not a flaw to be fixed; it's the nature of the tool. Building reliably means designing *around* these properties. Here are the four that bite, and what to do about each. ## It's non-deterministic - the same input can give different output **What it actually is.** A language model generates text by repeatedly picking a likely next token, and that pick involves some randomness. So the same prompt can produce different wordings - or different *answers* - on different calls. Expected behavior, not a bug. **The lever: temperature.** Most APIs expose a `temperature` setting (commonly a number from 0 upward) that controls how much randomness goes into each pick: - **Low temperature** → more focused and repeatable. Good when you want consistency: classification, extraction, "pick one of these options," anything where you'd be annoyed by variety. - **Higher temperature** → more varied and creative. Good for brainstorming, drafting, alternatives - anywhere you *want* different results each time. ```text low temperature higher temperature ─────────────── ────────────────── tends to the same, ranges more widely, safe, expected answer more surprising / creative use for: extraction, use for: brainstorming, classification, format-strict drafting, "give me options" ``` ⚠️ **Gotcha.** Low temperature makes output *more* consistent - it does **not** make it guaranteed-identical or guaranteed-correct. Don't build logic that assumes the model returns the exact same string every time. If your code needs an exact, reliable value, constrain the output shape (see structured output below) and validate it, rather than trusting the model to repeat itself. ## It can be confidently wrong - hallucinations **What it actually is.** A model can produce text that is fluent, authoritative, and *false*. It can invent a function that doesn't exist, cite a source that was never written, or state a "fact" with total confidence. This is a **hallucination**, and it's the single most important thing to internalize: the model is optimized to produce *plausible* text, not *true* text. Plausible and true usually overlap. Usually is not always. **What it does in real life.** The danger is precisely that wrong answers don't *look* wrong - no error, no flag, no lower confidence in the tone. A made-up API method reads exactly like a real one. 💡 **Key point.** Treat model output as a *draft from a fast, confident, sometimes-mistaken assistant* - never as a source of truth. Verify anything that matters: - For facts, prices, dates, or anything load-bearing - check against a real source. - For anything where a wrong answer causes harm (medical, legal, financial, destructive actions) - put a human in the loop, or don't use the raw output at all. - Show users where an answer came from when you can, so *they* can verify too. 🪖 **War story.** Plenty of people have shipped a chatbot that confidently quoted a refund policy, a discount, or a legal clause that never existed - and found out the business was on the hook for what the bot promised. The model wasn't malfunctioning; it was doing exactly what it does, generate plausible text. The missing piece was a guardrail around it. ## It can be slow - and it can fail **What it actually is.** A model call goes over the network to someone else's busy service and waits for text to be generated token by token - *slower and less predictable* than a typical database query or internal API. And like any network call, it can fail: the service can be briefly overloaded, rate-limit you, or time out. **What it does in real life.** Treat the call as what it is - a network request to an external dependency that is sometimes slow and sometimes down. The habits are the ordinary ones for any flaky external service: - **Set a timeout.** Don't let a single call hang your request forever. Decide how long you're willing to wait and give up past that. - **Retry transient failures - carefully.** For a temporary error (overloaded, rate-limited, timed out), waiting briefly and trying again often works. Back off between attempts and cap the number of tries, so a struggling service isn't hammered and your user isn't stuck. - **Don't blindly retry everything.** A "your input is too long" or "your key is invalid" error won't fix itself on retry - that's a bug to surface, not a blip to retry. Retry the transient stuff; fail fast on the rest. - **Have a fallback.** Decide what your app does when the model is genuinely unavailable: a friendly message, a cheaper backup model, a queued retry later. "It just spins forever" is not a plan. ⚠️ **Gotcha.** A naive integration with no timeout and infinite retries can turn one slow upstream moment into a pile-up that takes *your* app down - every stuck request holding resources while it waits. The timeout and the retry cap aren't polish; they're what keeps a model hiccup from becoming your outage. ## Asking for structured output (JSON) **What it actually is.** Often you don't want prose, you want *data* your code can use: a category, a score, a list of extracted fields. You can ask the model to reply in a structured format, almost always **JSON**, and then parse it. **What it does in real life.** You instruct the model, in your prompt, to respond with JSON in a specific shape - many providers offer a dedicated setting (a "JSON mode" or schema option) that strongly nudges or guarantees valid JSON. A simple extraction request might look like: ```json { "model": "some-chat-model", "messages": [ { "role": "system", "content": "Extract the order details. Respond ONLY with JSON matching: {\"item\": string, \"quantity\": number, \"rush\": boolean}. No prose, no explanation." }, { "role": "user", "content": "I'd like three of the blue mugs, and I need them fast." } ] } ``` And you'd hope to get back, in `choices[0].message.content`, something like: ```json { "item": "blue mug", "quantity": 3, "rush": true } ``` *What just happened:* You turned a sentence of free text into structured data your program can act on - no fragile string-parsing of prose. The system message both describes the exact shape *and* tells the model to return nothing but the JSON. ⚠️ **Gotcha.** Even with clear instructions, the model can still hand you something that won't parse - wrapped in a code fence, prefixed with "Sure! Here's the JSON:", or with a missing field. So **always parse defensively**: try to parse, and if it fails or a required field is missing, handle that case (retry once, fall back, or surface an error) instead of assuming the JSON is well-formed. Use the provider's JSON/schema mode when available - it makes valid output more likely - but validate anyway. ## Don't ship a foot-gun The difference between a demo and a feature is a handful of guardrails: - **Validate before you trust.** Check the output (parse the JSON, verify the facts that matter) instead of assuming it's right. - **Keep a human in the loop for anything consequential.** Especially actions that spend money, delete data, or give advice with real stakes. - **Cap the blast radius.** Timeouts, retry limits, output-length caps, and a fallback for when the model is down. - **Watch your spend.** Log the `usage` from Phase 2 so a runaway prompt doesn't quietly run up a bill. - **Never expose your key** (Phase 1) - server-side only, loaded from a secret. ## Recap 1. The model is **non-deterministic** - same input, possibly different output. Use **temperature** to dial randomness down for consistency or up for creativity; never assume identical results. 2. It can be **confidently wrong** (hallucinate). Treat output as a draft, not truth - verify anything that matters and keep humans in the loop for high-stakes cases. 3. It can be **slow or fail** - set timeouts, retry transient errors with backoff and a cap, fail fast on permanent ones, and have a fallback. 4. For data, ask for **structured output (JSON)** - and parse it defensively, because the model can still hand you something malformed. 5. The guardrails - validate, human-in-the-loop, cap the blast radius, watch spend, hide the key - are what make it shippable. You can now call a hosted model, reason about its cost and limits, and build around its rough edges. The remaining craft is getting the model to actually *do what you want* - writing instructions that produce good results reliably. > ⏭️ Next up: [Prompt Engineering, Plainly](/guides/prompt-engineering-plainly) - how to write prompts that get you the output you're after, without the cargo-cult.