The Conversational-AI Workflow Temptation

    |
    BySebastian Chedal

    The first time a conversational agent platform does something genuinely useful, it’s hard not to be impressed. You describe a workflow in plain language and watch it turn into a running system. There’s no GUI to configure, no YAML to write, no deploy cycle. You just say what you want and it shows up. If you’ve spent years wrestling with workflow tooling and code, that kind of frictionless experience is extremely exciting.

    I’m talking here about platforms where you use words to build workflows. So this applies to systems like OpenClaw, Hermes but also to Claude Code, Codex and to a degree also Claude Cowork and other systems, sometimes online with chat where you just use words to build things. So this also includes systems like Loveable.

    The truth is, as of this writing, the problems with conversation-only systems show up only later. Costs start climbing faster than usage can explain. An agent gives you a confident answer that turns out to be wrong. Skills and cron jobs pile up that nobody remembers setting up. Nothing’s actually broken. The platform is doing exactly what it was built to do. The issue is structural, and until model-harness providers can overcome it, once you know what to look for, the three main structural failure modes of self-building conversational agent platforms are pretty predictable.

    In this article:

    • Why LLMs often get deterministic questions wrong
    • How stale facts bake into agent definition files and become confident wrong output (context rot)
    • The five sprawl patterns that accumulate before anyone notices
    • Six engineering disciplines we’ve developed to address each type of failure
    • How to get the best of both worlds: Supervisory-engineering combined with conversational AI

    The silent-seduction of natural-language workflow creation

    Before we get into where this pattern breaks down, first we need to acknowledge how empowering conversationally-generated workflows is. Ben Vierck, the founder of Lit.ai says: “… a traditional GUI workflow design takes multiple steps of dragging, configuring, and connecting, while an LLM-driven platform just needs one sentence. Intent expressed in conversation carries across platforms, but GUI workflow skills rarely do. It’s just easier to describe a system in plain language than to build it in a visual tool someone else designed.” You don’t even need to be a designer, nor a programmer. This opens up all the creative opportunity to everyone.

    The compounding effect is real too. Changing a process becomes one more sentence instead of a 12-step redeploy cycle. A platform that writes its own skills can turn a conversation into an operational change in minutes. For mid-market teams without dedicated workflow engineers, that’s the difference between automation that actually happens and automation that stays permanently backlogged.

    The seduction lies in the immediacy. A conversational interface is meaningfully better than most of what preceded it, you think it, say it, it makes it. The question is not whether self-building platforms deliver value. They do. The question is whether these models are currently capable of building workflows that are robust, maintainable, cost effective and scalable with only a chat-box and an eager end-user at the helm.

    When a probabilistic system runs a rule with exactly one correct answer, sometimes it gets the wrong one

    Will Larson, a CTO who has led engineering at Stripe, Uber, and Calm, built an LLM-powered workflow to mark merged pull requests in Slack with a `:merged:` reacji. The workflow ran four steps: read the last ten Slack messages, extract the PR URL, check the GitHub merge status via API, add the reacji if merged. “This worked so well! So, so well,” he wrote. “Except, ahh, except that it sometimes decided to add `:merged:` to pull requests that weren’t merged.”

    The problem is not that the LLM was bad at reading Slack messages. It is that a merged/not-merged check is a binary fact with exactly one correct answer, and a probabilistic system sometimes gets binary facts wrong. What made this worse than a deterministic bug: the erroneous reacji destroyed the signal humans were relying on. “Erroneous additions of the reacji meant folks couldn’t evaluate whether to look at a given pull request in the channel based on the reacji’s presence.” The automation that was supposed to save time now required human verification on every result, which is worse than no automation. “Some problems are just vastly simpler, cheaper, and faster to solve with software,” Larson concluded.

    John Alistair Pritchard, whose engineering observations appear in Martin Fowler’s fragments feed, frames the generalization: most “agent” use cases are actually workflows (tasks with one right answer) not genuinely agentic problems requiring model judgment. “You need a function call” is his solution. A function call runs deterministically, is testable, goes through code review, and returns the same result on the same input every time. An LLM call for the same task is more expensive, slower, harder to audit, and occasionally wrong in ways that undermine trust in the whole system. Kent Beck, via the same feed, introduced the concept of the Genie Tarpit: the pattern of accumulating plausibly-working agents that require expert oversight to use safely. Each individual wish looks reasonable; the problem only becomes visible at the system level.

    Decision diagram titled LLM Call or Deterministic Script. A gray decision box at top asks whether the task requires genuine inference. Yes branch leads to an orange LLM call box describing judgment, inference, or ambiguity needed, with a blue outcome box stating variance in output is a feature, not a bug. No branch leads to a teal Use a script box describing binary rule, lookup, or fixed transform, with a teal outcome box stating one correct answer every time.

    Stale facts inside agent definition files produce confident wrong output, so context rot accumulates by design

    The second structural failure type is more subtle. Agent definition files, like the system prompts, skill documents, system messages, and configuration files that all tell an agent who it is and what it knows, are all written and edited by the LLM itself, without a person building them, so the snake starts eating its tail. This continuous loop often then leads to compounding context rot.

    This can lead to all sorts of problems, so let’s focus for now on just factual data in your identity files. Atlan named it as context rot in their agent sprawl research, and defined it as the accumulation of “stale, outdated present-tense facts in agent definition files” that produce confident wrong output without triggering a visible error. An agent configured last March with current pricing, an active partner list, or this quarter’s product roadmap will still cite those facts with full confidence in September. The agent is not broken. The agent definition is. And the agent has no mechanism to know the world changed.

    Atlan’s reporting on BNY Mellon found that the bank deployed hundreds of internal AI workspaces and 90% were abandoned within a month. The reason was not permissions or infrastructure failures, it was stale context. Agents that had been correct at build time were wrong when users relied on them, and correction required rebuilding agent definition files rather than adjusting a script or data record.

    NHIMG’s synthesis of Collibra’s analysis provides a useful framing: “LLMs operate on representations of systems rather than the governed systems themselves.” The representation was accurate at build time. The governed system changed. “Validate context at the point of use, not only at publish time.”

    Fountain City’s production answer to context rot is a state-modality documentation rule that bans present-tense facts from agent definition files entirely. The rule distinguishes three categories: IS facts (what is currently true — banned from definition files because they decay), SHOULD facts (durable behavioral directives — stable to include), and WAS facts (historical records that won’t change — stable to include). A definition file written under this rule reads like standing instructions rather than a current-events briefing. Agents built this way do not cite stale pricing, former partners, or outdated policy because the definition file never contained present-tense facts in the first place.

    Five sprawl patterns that tend to accumulate before anyone notices

    The third structural failure mode scales with platform adoption. When creating a new agent skill, cron job, or workflow requires nothing more than a conversation, the review step that normally sits between “I want this” and “this is running” disappears. The consequence: skills, crons, and workflows accumulate at the rate of conversations.

    Gartner projects that the average Fortune 500 will run more than 150,000 AI agents by 2028. Most of those agents will be built — and will continue building themselves — on conversational platforms. The quality of what they build will depend on whether anyone has built a proper harness or control-layer into their agent’s methodologies.

    IBM’s analysis of enterprise AI deployments names five recurring patterns:

    1. The skills junk drawer (a growing collection of skills nobody maintains and most people have forgotten exist)
    2. Shadow crons (scheduled jobs created conversationally that run without registry, documentation, or decommissioning paths)
    3. No registry (no central view of what agents exist, what they own, and who built them)
    4. Version drift (agent definitions change without version control, making rollback a memory exercise)
    5. No decommissioning path (agents built for a task that ended continue running indefinitely).

    Gartner, cited by IBM, found that only 13% of organizations say their AI agent governance is adequate. The difference with agent sprawl, compared to traditional automation-systems, is that the system can be confidently generating-wrong output from systems that appear to be running correctly.

    Cost bloat and flakiness are symptoms of system uncertainty, not the system being busy

    The most obvious effect of all this is increased cost and decreased reliability. The quick-fix is to solve it all with: stern words in chat to your AI, often baked with prompt-frustration “you are lieing!” or “try again and don’t make those mistakes again”, higher budgets, bigger models, longer context windows, more retries. But this instinct tends to make things worse.

    Alex Salnikov, in commentary on Structured Prompt-Driven Development, states: “Token consumption is mostly a symptom of uncertainty, not verbosity.”. What people should be optimizing for is instead: “fewer tokens per successful outcome”. Where success means no human review needed.

    A system that makes 500 tokens’ worth of API calls and reliably produces a correct answer is cheaper than one that uses 200 tokens and requires three retries and a human review to achieve the same result. Our cost optimization framework traces this pattern: cases where AI costs run high are almost always cases where the architecture is asking an LLM to do something deterministic, or where the context window is carrying stale or irrelevant information that increases uncertainty without improving output.

    Flakiness is a problem that is harder to catch because it doesn’t initially look broken. Larson’s reacji bot (above) did not crash, it returned an answer. The problem is just that the answer was sometimes wrong. With LLMs running workflow processes, you end up debugging a conversation transcript, not a stack trace.

    A stack trace names the line in code that fails. A conversation transcript shows you the message exchange; and then you have to infer where the logic went sideways. HBR, citing BetterUp and Stanford research, found that each instance of workslop (AI-generated output that seems plausible but requires correction) costs approximately two hours of human cleanup. In a high-volume agent system, two hours per instance compounds faster than the productivity gains accumulate.

    At scale, ungoverned self-building systems get canceled. Gartner, cited by Opsima from a June 2025 press release, predicted more than 40% of agentic AI projects will be canceled by end of 2027 due to escalating costs, unclear ROI, or inadequate risk controls. These cancellations are not happening because agentic AI does not work, they are happening because organizations deployed it without governance: no staging environment, no risk assessment, no IT approval workflow. For the security and access dimensions in enterprise environments, see our companion piece on AI agent security.

    How to fix and prevent conversational-AI workflow issues

    There are two ways through the problems, either cleanup and structure needs to be reactively and proactively applied to the conversational process, guardrails, guidelines in your AI Agent that help it build workflows in a more reasonable manner. Or engineering is needed to build the workflows, skills, agent identities and then conversations are used to execute the workflows rather than to build them. Below I break it down into 6 domains:

    Diagram titled The Self-Building Loop — Where It Leaks. A central blue box labeled Self-Building Agent shows it edits skills, workflows, and cron files. Three outward arrows indicate the three failure modes: left orange box shows LLM on deterministic rules leading to cost bloat and flakiness; top orange box shows stale facts in definition files leading to context rot; right orange box shows no reviewer gate between chat and production leading to skills and cron sprawl. A dashed container at the bottom shows six engineering discipline responses: LLMs to scripts for deterministic rules, human-own definition files and prompts, version-control prompts via SPDD framework, query live state with IS/SHOULD/WAS rule, schedule cleanup as first-class cron step, and per-agent exec allowlists with repo drift audits.

    1. Demote the LLM to a function call for deterministic work. If a task has exactly one right answer, like a status check, a price lookup, a permission evaluation, a date calculation. Don’t use LLMs, write a script. LLMs excel at intelligence; they tend to be overkill for binary facts. The LLM can draft the script in one shot, which preserves the conversational creation speed while eliminating runtime flakiness. We’ve applied this pattern when several pipeline stage gates that had been running as LLM-driven orchestration were producing unreliable results many times. You rewrite your workflow operations as deterministic Python scripts that inherited the same tools and trigger data resolve your inconsistencies. You can read more about this approach in our inside our pipeline writeup.

    2. Humans-own the build step. The platform can generate the skill, prompt, or workflow; the human commits it. Opsima’s Alon Gozlan describes the production pattern: “An AI agent designs the solution and writes the code. It tests everything… IT reviews, assesses the risk, and approves. Then it is live.” That review step takes days, not months, but nothing reaches production without it. This single constraint closes the gap between creation rate and oversight rate. A human owning the build step doesn’t even mean you are using manual time to re-write the code, it just means an engineer is owning the transition from conversation-processes to automated ones. Engineers should have the strong fundamentals ingrained in their methdology and know how to guide agentic development systems to refactor poorly structured workflows into highly deterministic stateful ones. Think Claude Code or Codex building OpenClaw skills and workflows, driven by an Agentic Engineer who owns the job.

    3. Version-control prompts and skills as first-class engineering artifacts. Structured Prompt-Driven Development (SPDD), articulated by Fei Zhang and Meng Xia at Thoughtworks, treats prompts as delivery artifacts with the same versioning discipline as code. Storing the code in GIT means you can version your prompts, roll them back if you need to and they get treated like everything else. If a prompt-backed feature gets a bug fix, the prompt updates in the same commit. All your artifacts stay synchronized and you no longer get the risk that two sources of truth start to drift apart.

    4. Query live state at inference time; don’t bake present-tense facts into definition files. None of your agents, including your agentic building systems, should be storing state in any of their memory or identity files. Why it is that model providers default to store state is beyond me, but until they fix this by design, it is up to us: implementors, to address this reactively. You need to educate your agents to adhere to this principle, and then you need to apply prevention and repair methods to scan for and clean up any states that are stored in memory or identity files like CLAUDE.md or AGENT.md files. Definition files should contain behavioral directives and historical records. Present-tense operational facts should be retrieved live at query time. Teach your agent where to look for values, not what the value is. This eliminates a serious source of context rot at the architecture level.

    5. Build cleanup as a first-class cron, not a reactive event. You want a script to empty the skills junk drawer, not when someone finally notices it is full. Fountain City runs a scheduled doc-hygiene crons that reviews definition files for various symptomatic problems… state stored in memory, code that should be a script stored as instructions in your LLM skill. This kind of repeated hygiene is similar to running a scan for file permissions, software updates and such. Meaning it should be part of your systems regular routine health check. Proactive weekly reports then start coming out your engineers can tackle by resolving the conversational-AI rot.  If you don’t do this, you will always be chasing the train as flames come running out the back. Our evaluation-led development guide covers how production failures can feed this loop directly: each incident becomes a regression test rather than a conversational patch that disappears in history. Signs you are doing this backwards are when your staff complains in frustration they keep having to tell their AI it is doing a bad job remembering how to do things, or it remembered things wrongly.

    6. Build paved-road infrastructure. The paved road is infrastructure that prevents individual engineering decisions from accidentally accumulating into systemic risk. Per-agent least-privilege exec allowlists mean an agent can only call the scripts and APIs explicitly approved for it, which contains the impact of any autonomous modification. Repo-as-source-of-truth deployments with drift audits mean the production state is always reconcilable against what the repository contains; if something is running that isn’t in the repo, the drift audit surfaces it. Our run exec allowlists applied to all our (and our client’s) agents mean that even a conversationally-building agent can only modify what its allowlist permits. Inadvertent skills can’t acquire permissions nobody intended to grant. You can do a lot with hooks, these can be set up to run additional code that improves, cleans or proactively builds skills through conversations the correct way, through determinism, not inference. Or your hook can pre-gate the operation and deny modification of a skill or identity file. There’s a whole mountain of opportunity for your AI engineers to craft a paves road that then allows your internal customers (the people using your agents day-to-day) to then build workflows conversationally with confidence they are being built correctly by design.

    Supervisory engineering is the operating model, not the training wheels

    You need to be deliberate about your oversight structure. Treating structure as design, not as an organic blank slate, waiting for models to improve.

    Mark Vella’s term for engineered structured-design is supervisory engineering: the middle loop between prompt-level debugging and hands-off automation. Supervisory engineering is a human-overseer role that helps to establish policy and owns delivery. Engineering approves what reaches production, reviews what the platform generated, builds paths for the AI to follow and systems to audit and maintain correct deisgn behavior. This closes the gap between creation rate and oversight rate through structural constraints, rather than continuous reactive manual attention.

    Waiting for improvements in the model so far have had little effect. More powerful AI just seem to fail more confidently. The structural failure modes described above where LLMs run deterministic rules, store stale context in definition files, and build inappropriate or unauthorized skills without review, are not model capability problems. They are engineering-discipline problems (and often a lack of any engineering at all…).

    Fountain City builds and operates autonomous agent systems for clients across industries. The disciplines described in this article are applied to our own production fleet and to every client deployment. While they are not heavy process, these handful of structural constraints create a self-building system that positively compounds instead of negatively accumulating. If you’re working through the governance and operational side of a deployment, our managed autonomous AI agents page covers how we approach this work with clients.

    Ornate classical fountain rendered in fine ink etching style, with water jets arcing from tiered stone basins in precise hatched lines

    Frequently Asked Questions

    What is agent sprawl?

    Agent sprawl is the unmanaged growth of AI agents, skills, workflows, and cron jobs in environments where creation is easy and review is optional. The term extends the same patterns that affected RPA deployments, microservices architectures, and SharePoint environments. The distinctive element in the agent context is that the failure mode is confident wrong output from systems that appear to be running correctly, rather than broken connectors or unmaintained bots. Gartner projects the typical Fortune 500’s AI agent count will grow from dozens today to six figures by 2028, making sprawl governance a first-order operational concern rather than a future problem.

    What is a self-building AI agent platform?

    A self-building AI agent platform is one where agents running on the platform can modify their own skills, create new workflows, write new cron jobs, and update their own definition files through conversation — without a traditional deploy cycle. OpenClaw, Hermes, and similar platforms in this class enable conversational creation of operational automation. The benefit is real: creation time drops from hours to minutes. The structural risk is that creation rate outpaces oversight rate by default, and the three failure modes described in this article accumulate without deliberate engineering discipline.

    Why do AI agents built through conversation tend to break in production?

    Three structural mechanisms account for most production failures. First, probabilistic models sometimes get deterministic questions wrong, and occasional wrong answers destroy the reliability of automation entirely. Second, definition files written at build time carry facts that decay: partner lists, pricing, policy details. The agent does not know these have changed, so it continues citing them confidently. Third, conversationally-created skills and cron jobs skip the review gates that catch design problems before they reach production. Per research cited by NHIMG, 80% of organizations report their AI agents have already acted beyond their intended scope. All three failure modes are predictable; none require model failure.

    When should an AI agent workflow be an LLM call versus a deterministic script?

    Use an LLM call when the task requires genuine inference: parsing unstructured input, handling ambiguous edge cases, reasoning across incomplete information. Use a deterministic script when the task has exactly one right answer: status checks, data lookups, arithmetic, permission evaluations, binary decisions. The practical test is Larson’s: if the correct output doesn’t change when run on the same input, an LLM is unnecessary and introduces flakiness. LLM-driven prototyping to discover the right logic, then script conversion for the production path, is the pattern that tends to work — and the LLM can usually write the script in one shot.

    What is ‘context rot’ in agent definition files, and how do you prevent it?

    Context rot is the accumulation of stale present-tense facts in agent definition files — the prompts, system messages, and skill documents that define an agent’s behavior. The facts were accurate when written; by the time the agent cites them, they may not be. The agent has no mechanism to detect the gap. Prevention happens at the writing level: ban IS-category facts (currently true statements) from definition files and replace them with either live-lookup logic (retrieve from a live source at inference time) or durable behavioral directives that don’t depend on present-tense facts. Per NHIMG’s analysis, the discipline is to validate context at the point of use, not at publish time.

    How do you review AI-generated skills, prompts, and cron jobs at the pace they’re being created?

    The key is structural review rather than manual attention. Manual review can’t keep pace with conversational creation. Structural review means the platform requires human approval before the deploy step runs — the AI generates, the human commits. Opsima describes the pattern for governed agentic deployments: “IT reviews, assesses the risk, and approves. Then it is live.” It takes days, not months, but nothing reaches production without the gate. Combined with a skills registry and a scheduled decommissioning process, this keeps the portfolio auditable without requiring continuous manual scrutiny.

    Is stronger engineering governance actually the slow path, or does it make agents faster?

    Engineering governance tends to make agent systems faster at the portfolio level, though it adds overhead at the individual build level. Ungoverned systems accumulate debt: duplicate skills, conflicting crons, stale definition files that require debugging before the next build can use them. Each piece of debt slows the next build. Governed systems are auditable, which means new skills can reuse existing components rather than duplicating them, and definition file updates happen systematically rather than after a production incident. Opsima’s direct observation: governed agentic AI “is not slower than ungoverned AI. It is the only kind that reaches production safely and stays there.”

    What is Structured Prompt-Driven Development (SPDD) and where does it fit?

    Structured Prompt-Driven Development, articulated by Fei Zhang and Meng Xia at Thoughtworks, is the discipline of treating prompts as first-class delivery artifacts — versioned in the repository alongside the code they work with, updated in the same commit as any related code change, reviewed through the same pull request process. The fix-the-prompt-first rule is the operational guide: if a behavior is wrong, address the prompt before adjusting the model or adding retry logic. SPDD fits as the engineering discipline that closes the gap Perviz identified: “The reasoning lived in a ChatGPT tab, not in the repository.” When reasoning lives in the repo, the full system is auditable and rollback is a git operation.

    How is agent sprawl different from earlier waves like RPA sprawl or microservices sprawl?

    The mechanics of accumulation are the same: easy creation outpaces governance by design, and the resulting portfolio becomes unmanageable. The distinctive difference is the failure mode. RPA sprawl produced technical debt — brittle bots that broke visibly when schemas changed. Microservices sprawl produced ownership complexity — systems with no clear accountable team. Agent sprawl produces confident wrong output — systems that appear operational, return plausible answers, and are wrong in ways that don’t trigger visible errors. That confidence makes the problem harder to detect early: there is no broken connector or failed deploy to alert you. By the time you notice, the stale context and shadow crons have been running for months.

    What does ‘fewer tokens per successful outcome’ mean as a cost metric?

    It refocuses cost analysis from consumption to efficiency. Token consumption per call is easy to measure but not the right signal: a system that uses 500 tokens and reliably produces a correct answer is cheaper than one that uses 200 tokens and requires three retries and human review. Salnikov’s framing — “token consumption is mostly a symptom of uncertainty, not verbosity” — means that when costs run high, the root cause is that the system is uncertain about the right answer. Reducing uncertainty through clearer prompts, deterministic substitutions for LLM calls where tasks are binary, and better context hygiene tends to reduce cost more reliably than optimizing token count directly. Our cost circuit breaker post covers a concrete production case of this substitution pattern.