AI Cost Control That Doesn't Break Velocity

I learned this lesson the expensive way. When I first started building AI-powered workflows, I watched costs spike in ways that seemed random. One week would be cheap, the next week would blow the budget. It took months of digging through logs to understand what was actually happening.

The pattern was consistent: most of the cost wasn't coming from the initial calls. It was coming from repeated retries because the status UX was unclear and users kept re-triggering operations. It was coming from regenerating outputs instead of importing existing ones. It was coming from sending the same prompts repeatedly because nothing was cached. And it was coming from storing outputs poorly so they couldn't be reused.

The Three Levers of AI Cost

After debugging too many cost spikes, I've come to see AI cost through three levers: reduce calls, reduce tokens, and reduce waste. Each lever has different techniques, and the combination of all three is what keeps costs manageable at scale.

Lever 1: Reduce Calls

The most expensive call is the one you didn't need to make. Before every AI operation, ask three questions: Can we fetch an existing result? Can we import a downloaded artifact? Does the output already exist?

This sounds obvious, but it's remarkable how many systems lack these pathways. Users trigger regeneration because there's no way to fetch what was already generated. Operations run twice because the first result wasn't stored in a discoverable way. The "just run it again" pattern becomes the default, and costs multiply.

Lever 2: Reduce Tokens

Keep prompts as small as they can be while still producing good outputs. This means avoiding dumping entire document trees into a single prompt. Pass references instead of full text when the model can work with references. Summarize once and then reuse that summary rather than re-summarizing every time.

I've seen prompts that include thousands of tokens of context that the model never actually uses. The context is there "just in case," but it adds up fast. Being disciplined about what goes into a prompt requires more design work upfront but pays off in every subsequent call.

Lever 3: Reduce Waste

Waste is output that isn't stored, output that isn't readable, and output that can't be debugged. If your system can't reuse outputs, you'll pay for regeneration forever. Every output should be stored, normalized, and accessible for reuse.

Budgeting That Actually Works

The simplest reliable budget system is per-run caps. Instead of trying to predict monthly costs and getting surprised, set explicit limits at the operation level: "This workflow run can spend at most X units." "This step can spend at most Y units."

If you exceed the cap, the system should stop the run, mark the step failed with recovery instructions, and allow import or fetch-by-id as a cheap recovery path. This turns cost control into a deterministic behavior rather than a monthly surprise. You know exactly when and where spending will stop.

Caching: The Underrated Superpower

Cache at the right granularity. Deterministic transforms like normalization and parsing should always be cached, you'll get the same result every time, so there's no reason to redo them. Stable inputs like summaries should be cached because they change rarely. Fetch-by-id results should be cached because the external system is the source of truth.

Avoid caching anything that depends on secrets or user-private context unless you isolate by tenant or user. This sounds obvious but it's easy to accidentally cache something that contains PII or credentials, creating both a security risk and a correctness bug when the wrong data gets served to the wrong user.

The Import-First Pattern

If you can import outputs, you get three benefits that compound. You don't rerun steps when something fails to save, you just import the result that was generated. You can re-materialize documents without regenerating content. And you can debug without repeating the expensive operation.

This is why I always build import pathways alongside generation pathways. The import path handles failures gracefully, reduces cost, and improves debuggability. It's more work to build both, but the payoff is enormous in production systems that need to stay reliable and affordable.

A Cost Checklist for AI Steps

Before shipping any AI-powered step, I run through a checklist that ensures cost control is built in from the start. This prevents the "we'll optimize later" trap that usually means "we'll spend 3x more than necessary until someone notices."

Trade-offs

Strict caps can make the system feel "fragile" until recovery paths exist. Users will hit the cap and get stuck if there's no way to proceed without regeneration. This is why import-first is so important, it's the escape hatch that makes strict budgets workable.

Caching can introduce stale behavior if you don't key caches properly. A cache that returns outdated data is worse than no cache at all because it adds debugging complexity on top of the original problem. Cache invalidation strategy needs to be designed upfront, not bolted on later.