Production is a process, not a prompt

The demo is the easy meeting. Someone has wired a model to a couple of tools, it does something impressive on a clean example, and the room leans forward. I work in marketing analytics, so I have been in that room from the side that has to live with the result: the Monday report, the pacing check, the "why did this number move" question that shows up while a stakeholder is already screen-sharing.

An agent earns a place in that week only if it is still useful when the data is late, a campaign got renamed, and the person who built the demo is in another meeting. That condition is what I mean by production. A strong prompt is the starting point. The operating cadence around it is the part that determines whether the thing is still running next quarter.

This is the way I approach it.

Start from the decision, not the model#

Before I care which model is in the loop, I want three names written down.

The workflow. A recurring job a person already does, with a cadence and a definition of done. In a marketing analytics org that is often a narrative someone has to produce (what changed, which campaigns were responsible, what we are doing about it), a QA pass on tracking and names, or an exception list for pacing and broken routing. The common trait is that the work is frequent, it has a source of truth, and it hurts when it is late or wrong.

The system of record. Where a skeptical analyst would go to check the answer. The warehouse, the ad platform, the CRM, the semantic layer the team already trusts. If the agent needs a side spreadsheet of "what it thinks the numbers are," it will drift, and you will not know which number a stakeholder quoted.

The owner. The person who feels the pain if the workflow fails, and the person who can turn the agent off. On a small team that can be one person. It cannot be "the AI committee."

If any of the three is missing, I have a prototype looking for a sponsor. Those are fine to build on a weekend. I do not roll them out under a company name.

In my world the first candidate is usually something like the Monday narrative. The inputs already live in a warehouse or an ad platform. The output is a note a person is willing to send. That is a better first agent than an open-ended analyst that can "look at anything."

Put the agent next to the stack you already have#

The architecture I will defend in front of IT is boring. That is a feature.

Data stays where it is. The agent reads through an allow-list: a semantic model, a saved query, a documented API. "Answer questions about performance" is a wish. "Read this approved model, for this date range, at this grain" is a tool. Arbitrary SQL against production, with the model picking the joins, is how you end up in an incident review explaining a number you cannot reproduce.

The runtime is a replaceable box in the middle. It calls tools, it drafts an artifact, it stops. It does not become a new system of record, and it does not keep a shadow copy of the business. When the warehouse is restated, the next run should reflect the restatement without anyone remembering to update the agent.

Writes are a different product. The first version I will ship is almost always a draft a human sends: a note, a ticket description, a summary in the channel where the team already works. Write-back into an ad account, a CRM, or a budget comes later. It goes through an API the company already uses, with an identity that is a service account rather than a borrowed employee session, an approval where the action is hard to undo, and an idempotency key so a retry does not double-spend.

Around that, two things stop being optional once anyone else depends on the run:

• An audit log of what it read, what it proposed, and who approved or edited the result. If I cannot replay Tuesday, I cannot change the system on Wednesday.
• A policy gate in front of anything that leaves the building or spends money. The model does not get to decide that a policy is annoying.

sources (warehouse, ad platforms, CRM)
        |  read-only, through approved models and APIs
        v
   tool layer (explicit allow-list)
        v
   agent runtime (drafts an artifact, then stops)
        v
   policy gate ------> audit log
        |
        +--> draft a person edits and sends      (ship this first)
        +--> write-back through an existing API  (later: approval + idempotency)

Nothing in that picture is a reason to wait on a perfect platform. It is a reason to draw the picture before a framework makes the demo feel finished. Frameworks change. The allow-list and the audit log are the part the company is actually buying.

Roll it out in an order you can reverse#

I use three phases. A good meeting does not promote me past the one I am in.

Shadow#

The agent runs on the real cadence, against real inputs, and a human still produces the real deliverable the way they did last month. We compare. The agent is not in the channel yet. This phase is where I find out that the workflow depended on a private spreadsheet, a naming convention nobody wrote down, or a data issue the demo never hit. Shadow mode is cheap. A wrong autonomous action costs more than another week of comparison.

Draft#

The agent's output becomes the starting point. A person edits and sends, and their name is still on it. What I measure is dull on purpose: how long the edit takes, what they keep changing, and whether a stakeholder asked a follow-up the draft could not support. A feeling that the model is smart is not a metric. A shrinking edit, on ugly weeks as well as clean ones, is.

The hours the team gets back show up in this phase. I would rather stay in this phase for anything customer-facing or finance-facing than skip ahead because a pilot channel applauded.

Limited autonomy#

A slice that is easy to reverse, small in dollars and in reputation, with a cost cap and a kill switch. The old process stays in place until this slice has survived a few bad weeks, including the week a source was late. Autonomy is earned on a narrow path, after the draft phase has a track record.

Evaluation is a weekly habit#

A launch checklist is not an evaluation program. I keep a small set of golden tasks taken from real weeks, and I refuse to let the set be all happy paths. Partial data. A renamed campaign. The month finance restated a number. The platform that was down on Thursday. If the golden set is only the demo script, the score will be flattering and useless.

Once the workflow is live, someone reads a sample of outputs on a schedule. Early on that schedule is weekly, and the reader is the owner. The failure I actually worry about is quiet drift. The job is still green, the prose is still confident, and the answer is slowly wrong because a column changed or a filter got dropped. Confidence is not a signal I will trust. A diff against the system of record is.

The human's edited version goes back into the pile. That pile is the regression set: real inputs, real tool calls, and the artifact a person was willing to put their name on. When I change a prompt, a tool, or a model, I replay that pile before the change hits the next Monday. If I cannot replay it, I have a chat window with a deploy script.

Answer the security questions before the rollout meeting#

Legal, IT, and procurement ask the same things every time. I would rather write the answers into the design than perform them at the end.

What leaves the network. Which fields, to which provider, under what retention terms, and whether the arrangement allows the provider to train on it. If the honest answer is "whatever the agent decided to paste into the prompt," the tool layer is not finished. The outbound payload should be boring and predictable.

How identifiable the data is. My default is aggregates and campaign-level facts. Row-level customer data is a different review, a smaller allow-list, and often a reason to cut scope. A sentence in the prompt that says "be careful" is not a control. Access is a control.

Which identity it uses. A service account with the minimum read it needs, and no standing write unless we are in that later phase. A personal login, or a token copied off a laptop because the demo needed it, will outlive the person who pasted it.

Who is accountable when it is wrong. The audit log is for that conversation. So is the named owner. "The model said so" is not a cause of record I am willing to bring to a stakeholder.

None of this requires a two-year platform program. It requires the first workflow to be narrow enough that the answers fit on a page.

Give it an owner, a budget, and a fallback#

An agent without an owner is a script with a better pitch. I want a name next to it in the same place the team tracks the rest of the recurring work: who gets the alert, who reviews the sample, who is allowed to widen the allow-list.

It also needs a budget that is literal. Token spend and tool calls are a cost center whether or not anyone has opened the invoice. A cap, an alert, and a glance in the weekly ops review are enough. I have yet to see a team regret knowing what a workflow costs per run. I have seen teams get surprised.

The fallback is the old process, which stays available in week one. The point of the early phases is that turning the agent off is dull. Someone does the job the way they did it last month. If turning it off would be an emergency, it went into production too early.

Show the workflow in a minute#

A thread is a bad place to explain a click-path, and a long tour is a bad file to commit. When seeing the screen is faster than describing it, I record the path in OBS, export a short compressed MP4, and drop the file next to the paragraph it supports. One pass is enough: the source, the draft, the edit I actually made. If I cannot tell that story in about a minute, I am touring the tool instead of showing the decision.

I keep these small. 720p or 1080p, compressed, short enough that I would rewatch it. A raw OBS capture is not the file that belongs in the repo.

The player below is the pattern. The file checked into the site is a two-second stand-in, so the page has a real clip to load before I have a recording worth keeping. Replacing it later is a file swap. A real one for this essay would be the shadow-mode review: the agent's draft, the version a person sent, and the diff that tells me whether the edit is shrinking.

The whole method#

Pick one workflow a named person already owns. Read from the system of record through a narrow tool, and keep the agent from becoming a second copy of the business. Ship a draft a human still sends. Keep the ugly weeks in the evaluation set. Widen autonomy only on a slice you can turn off without a war room.

The model will change twice before the process does. Build the process so the model can.