The gap between an impressive GPT-4 demo and a reliable production LLM integration is wide, expensive, and full of embarrassing surprises. We use language models for specific jobs in our own products, such as reading receipts in PaidUp and writing fix requests in DevPin. These are the lessons we find matter most.
Hallucinations Are a Product Problem, Not Just a Model Problem
Every LLM will occasionally produce confident, plausible-sounding nonsense. The question is not "how do we eliminate hallucinations?" because you can't. The question is "how do we design our product so hallucinations either don't reach users, or are harmless when they do?"
The answer is to check the output before anyone relies on it. Ask the model for a fixed structure, check that the answer fits it, and fall back to a plain default or a person when it does not. Where the result matters, show it to a person to confirm, the way PaidUp lets you check every receipt line before saving.
Latency Will Surprise You
A hosted model can answer in under a second one minute and take many seconds the next, depending on load and how long the answer is. That is routine variance, not a rare edge case. If you build a product that requires LLM responses in a synchronous request-response flow, you will have a bad day at 9am on a Monday.
- Use streaming responses for anything user-facing so rendering starts as tokens arrive
- Move non-urgent LLM calls to async background jobs with a queue
- Set aggressive timeouts and always have a graceful degradation path
- Cache aggressively, since many prompts are semantically identical; hash and cache responses
- Consider smaller, faster models for latency-sensitive paths (GPT-4o-mini is remarkably capable)
Prompt Engineering Is Engineering
Prompts are not documentation; they are code. They should be version-controlled, tested against regression datasets, and reviewed before deployment. Treat every prompt change as a code change: review it, test it against known examples, and keep a way to roll it back.
A small wording change to a prompt can make results noticeably worse in ways nobody spots by reading it. Test sets are not optional.
Cost Management at Scale
LLM API costs scale faster than you expect. A feature that costs $0.002 per user action seems negligible at 1,000 users. At 500,000 users doing that action 10 times a day, it's $10,000 per day. Track the cost of every call from day one, set limits per user, and use smaller models wherever quality allows.
The Eval Suite Is Your Safety Net
An eval suite is a curated set of inputs with expected outputs (or output quality metrics) that you run before every deployment. It's the closest thing the LLM world has to unit tests. Building one takes time upfront but pays back immediately, and it's the only way to catch regressions before they reach production.
Start with 50 representative examples from your actual use case. Label the expected outputs. Run evals on every model change, prompt change, and API version upgrade. Expand the suite every time a production bug is found. After six months, you'll have an eval suite that gives you genuine confidence in your deployments.