Learnings from building reliable agentic systems that make it into production for enterprises

I opened my workshop at The AI Conference with a few questions. Who had built an agent? Almost the entire room raised their hands. Who had deployed one into production? Fewer hands. Who had scaled it to tens of thousands of users? Three or four remained.

That gap was what I wanted to spend the session on. Getting an agent to do something useful once is exciting. Getting an enterprise to depend on it means answering a different set of questions: what can it access, what happens when it fails halfway through, and how do we know a change made it better?

Leading the open source team at CrewAI, I get to work on the framework behind these systems and see how teams use it. Across the different use cases we discussed, the same architectural lesson kept coming up: reliable agentic systems balance determinism and agency.

Here are the lessons I wanted people to take home from the workshop.

1. Decide where reasoning earns its place

We used lead qualification as an example. Fetch a lead from a CRM, score it against a rubric, and decide what to do next.

Fetching a record by ID is a normal API call. Comparing a validated score against a threshold is a conditional. Neither needs an agent to reason through it. Scoring an ambiguous company profile against a qualification rubric, or researching whether that company is a good fit, is where reasoning can add value.

In CrewAI, we use Flows as the orchestration layer around that work. The important design choice is deciding which steps your application controls and which decisions you delegate to a model.

This makes the system easier to reason about. If the rule is “research leads whose score is greater than seven,” code can enforce that rule after validating the score. The score itself is still a model judgment. Putting a conditional around it gives you a predictable transition, but you still need to evaluate whether the scoring is any good.

I find this distinction useful whenever a team is designing an agentic workflow: identify the uncertain decision, give the agent enough context to make it, and keep the surrounding business rules explicit.

2. The harness deserves as much attention as the model

A model operates inside a system. The tools it can call, the context it receives, the state you preserve, the hooks around its actions, and the conditions for stopping all affect the result. That's what I mean by the harness.

During the workshop, I shared our experience evaluating a coding harness on Terminal-Bench 2. We saw scores move from roughly 20% to above 60% using the same model as we changed the surrounding architecture and iterated on the harness.

That was a result from a particular benchmark and setup. It doesn't tell you what improvement to expect on your own workload. What made it useful to us was having a measured way to see whether changes to the harness helped.

One of those changes was moving toward a plan, execute, observe, and replan structure. There was still reasoning and tool use inside it, but progress became explicit state.

A useful plan needs more than a list of intentions. Each step should describe the work, its dependencies, its status, and what counts as done. A worker executes a step. An observer checks the result against the expected outcome and determines whether the plan still makes sense. Code can select the next ready step and enforce the execution limits.

One example from the session was a missing dependency in a clean sandbox. The tool call fails. The observer can recognize that the environment needs preparation, and the plan can change to include installing the dependency before trying again.

The recovery has to be bounded. Set limits on retries, replanning, iterations, and elapsed time, with a defined outcome when those limits are reached. A workflow that keeps trying indefinitely can consume a lot of tokens without making progress.

3. Put enforcement at the action boundary

We talked about guardrails, hooks, and human intervention as different ways to control execution. They solve different problems.

A programmatic check can enforce an allowed domain, a required field, or whether a user has permission to call a tool. A model-based judge can assess something more subjective, such as whether a report meets a quality rubric. Hooks give you places to inspect or intervene around model calls and tool calls.

For enterprise applications, I want hard access rules enforced by code and by the services behind the tools. If a tool only needs to read records, give it read access. If an operation requires approval, check that approval before the operation runs. A prompt can describe the policy, but the application needs to enforce it.

We demonstrated this with a conversational workflow that had two profiles. A viewer could fetch information. An analyst could also perform web research. The model could route a request toward research, and the application would check whether the active profile was allowed to take that route.

That separation matters. A router chooses a likely intent; authorization determines what the user may do. Keep the permission check in the execution path so a mistaken route cannot grant access.

Context needs the same care. Give an agent the information its task and user are allowed to access. Hooks can help inspect or sanitize messages, but access controls should also apply when data is retrieved. Once sensitive information has reached the model, filtering its final response is too late to prevent that exposure.

4. Human intervention is most useful before a consequential step

Human in the loop often gets implemented as someone reviewing the final answer. In the workshop, I pushed for placing it at the points where a person can still change the outcome.

In the lead qualification example, fetching the CRM record probably doesn't need a reviewer. An uncertain qualification decision might. Sending an external campaign could be another point where approval makes sense, depending on the workflow.

The useful question is where human judgment changes the next action. At that point, show the reviewer the proposed action, the evidence behind it, and the decision you need from them. Preserve enough state that the workflow can continue after they respond.

Checkpointing supports this too. If a workflow has already spent an hour collecting and analyzing information, a pause or failure shouldn't automatically mean repeating all that work. Persist the state needed to resume from a known point.

We also discussed forking from a checkpoint. If a qualification rubric changes, you might reuse the lead data and rescore it against the new rubric. You still need to decide whether that saved data is fresh enough for the new decision.

A practical consequence is that recovery needs to account for side effects. Restoring workflow state doesn't undo an email that was sent or a ticket that was created. Track completed actions and use idempotency keys where the destination supports them, so retrying a step doesn't repeat an external action. The AWS Builders' Library has a useful explanation of making retries safe with idempotent APIs.

5. A conversational interface can still have explicit rules

One of my favorite parts of the workshop was showing that a controlled workflow can still feel conversational.

A person can ask for research in ordinary language. The system can classify the request, check their permissions, run the appropriate work, and return a response. If they lack permission, the application can produce the denial message directly. That message doesn't require a separate model call, although classifying the request may already have used one.

This also makes routing worth measuring on its own. If the router only needs to choose between an order request, a return request, and a clarification, evaluate models suited to that decision. Measure routing accuracy alongside cost and latency.

We explored a specialized classification model during the session. The broader lesson was to choose the model for the job each step needs to do. A routing decision and a research report have different requirements.

Give the router a clarification path too. If the request is ambiguous or outside the supported routes, asking a useful follow-up can be the correct outcome. When using confidence scores to make that decision, choose thresholds based on evaluation examples. A score is useful only to the extent that it predicts correct routing on your workload.

6. Traces should help you make the next engineering decision

An audience question captured a familiar frustration: the same input produces the right decision one day and the wrong one another day. How do you find out what happened?

You need a record of the execution: the context provided, tool calls and their results, selected routes, validation failures, retries, and the point where the workflow stopped. Track cost and latency at useful boundaries too. Handle sensitive data deliberately when deciding what to retain in those traces.

That evidence helps you narrow the problem. Did a tool reject its arguments? Did retrieval return the wrong context? Did the planner miss a dependency? Did the workflow exhaust its retry limit? A final answer alone can't tell you those things, and a trace still needs investigation to establish the cause.

Use those signals operationally. A workflow that normally finishes in thirty seconds and suddenly takes five minutes should have a timeout, an alert, and an owner who can respond. The same engineering habits that keep other applications running apply here.

Traces are also useful during development. In our benchmark experiments, we fed execution traces back to a coding agent to help revise the harness's internal prompts, then reran the benchmark to see whether the changes improved the score or caused regressions.

The evaluation step is what closes that loop. A plausible prompt revision is still a hypothesis until you measure its effect.

7. Define what good means for your business workflow

The questions at the end of the session kept returning to evaluation. What should a golden dataset contain? How do you know an improvement on a general benchmark helps a specialized enterprise use case?

A public benchmark can tell you something useful about the capabilities it tests. Your production workflow needs its own examples and acceptance criteria.

For lead qualification, I would start with representative leads, a versioned rubric, and decisions reviewed by people who understand the business. Include ambiguous profiles, missing information, tool failures, and cases that should be escalated. Those examples let you evaluate the scoring and the behavior around it.

Decide what success means before comparing changes. You might check whether the system selected the correct route, respected permissions, produced the required evidence, and completed within a cost and time budget. Some checks can be exact assertions. Others need a rubric or human review. Repeat runs where variation matters, and keep a held-out set so you're testing whether improvements generalize beyond the examples used to tune them.

Version the data, rubric, prompts, tools, and model configuration together. When an outcome changes, you need enough information to investigate what changed with it.

What I would check before shipping

Before putting an enterprise workflow in front of users, I'd want concrete answers to these questions:

• Where does reasoning add value? Identify the decisions delegated to agents and the rules enforced by code.
• What can each user and agent access? Check permissions at retrieval and action boundaries.
• What counts as done? Define step outputs, validation, and completion criteria.
• What happens when progress stops? Set execution limits and an escalation path.
• Where can a person intervene? Place review before the actions that need it.
• Can execution resume safely? Persist state and account for external actions that already happened.
• Can someone diagnose a failed run? Capture useful traces, establish alerts, and assign ownership.
• How will we detect a regression? Evaluate changes against representative cases with agreed acceptance criteria.

The gap between the hands raised at the start of the workshop is where much of the engineering work lives. Models give us useful reasoning capabilities. Our job is to give that reasoning a place in a system we can operate: explicit state, enforceable rules, bounded recovery, and evidence that the workflow does what the business needs.

That's the work I want teams to spend more time on as they move their agents into production.

---

This post draws on Lorenze Jay: Designing Reliable Agentic Systems that make it into Production, my workshop recorded at The AI Conference. The recording includes the hands-on examples and audience Q&A behind these lessons.