Ship a Degraded Mode With Every LLM Feature
Before an LLM feature goes live, decide what the product does when the model is slow, down, rate-limited, or wrong. Timeouts, failover, cache, templates, flags, and how you measure the fallback.
Index
Essays on retrieval, evals, agents, and the product work around production LLMs.
Before an LLM feature goes live, decide what the product does when the model is slow, down, rate-limited, or wrong. Timeouts, failover, cache, templates, flags, and how you measure the fallback.
A design review that ends with "let's sync again" is not unfinished thinking. Here is how I write the question, name options, timebox, decide who decides, and leave with a written next step.
A design doc that never forces a choice is theater. Here is how I write ones that decide: when to write, how to frame the problem, and how to run review so it ends.
Users feel LLM latency before they feel model quality. Here is how I set a per-feature budget, spend it on first token and streaming, and watch p50 and p95.
A numbers-literate guide to predicting an NYC Marathon finish time from race formulas, long runs, and the five-borough course, then pacing it honestly.
A tour and a stack of docs is not onboarding. Here is how I prepare before day one, pick a scoped first change, pair without taking the work, and make week one a team outcome.
Shipped prompts drift and break silently. Here is how I version them next to the calling code, review diffs, gate on evals, pin models, and roll them back.
Closing a ticket is not owning the outcome. Here is how I define done, own rollout and cleanup, close loops early, and share the work without hoarding it.
Prompt injection is untrusted content in the model's context. Here is a practical playbook: isolate data from instructions, starve tools of privilege, gate writes, and test the attack path.
Vague praise and late notes do not change the next pull request. Here is how I give feedback engineers can actually use: specific, timely, and followed up.
Human-in-the-loop only works if people actually read the ask. Here is how I decide what needs approval, show the real action, and treat a no as final.
A blocked teammate burns more hours than your interruption. Here is how I notice stuck work, unblock it without taking over, and still protect focus time.
Treat the schema as the product contract. Here is how I design typed LLM outputs, validate and repair them, version the shape, and eval field by field.
Mentors invest in people who make help cheap to give. Here is how I arrive prepared, ask about decisions, close the loop, and turn advice into a pattern.
Tool schemas are prompts. Here is how I name tools, shape parameters, return compact results, write recoverable errors, and eval whether agents can use them.
Code review is the highest-leverage teaching loop on an engineering team. Here is how I review in a way that grows the author, not just the diff.
A long context window feels like memory, but it is just a working set. Real memory needs selection, persistence, and a plan for what to forget.
A practical framework for building retrieval and answer-quality evaluations that catch real regressions instead of producing vanity metrics.
Why prompt caching changes latency, cost, and product design decisions — and why teams should treat it as a user-facing capability.
A practical view of agent observability: what to log, what to surface, and what teams need to understand before agents become trustworthy in production.
Why AI search changes how brands earn visibility, and why the next era of search is less about ranking pages and more about becoming a trusted source.
A practical way to think about model routing across cheap, fast, and strong models without turning your stack into a chaotic science project.
Why the Model Context Protocol matters: it gives agents a cleaner way to discover tools, carry context, and interact with software systems.
RAG is powerful, but it is not a default answer to every AI product problem. Here are the cases where I would avoid it or simplify it.
Synthetic data can speed up AI evaluation pipelines, but only if it is used with strong guardrails and real-world calibration.
Why AI browsers matter: they collapse reading, searching, summarizing, and acting into a single interface.
Speed matters in startups, but shipping fast only works if users still trust what you launch.
A personal look at what it’s like working at Evertune as a software engineer in NYC — the people, the collaboration, and what makes it different.
A practical look at how Sora changes video creation and how teams can use it to design cameos, campaigns, and brand storytelling.
My fascination with artificial intelligence didn't begin in a traditional computer science lecture. it started in a classroom alive with sound. At Northwestern University, I enrolled in COMPSCI 352: M...