Brokers as scaffolding for recurring duties.

One of my items/curses is an limitless fixation with how processes will be optimized.
For a short second early in my profession, that was targeted on enhancing how people collaborate,
however that shortly switched to determining how we will reduce human involvement, and remove
human-to-human handoffs as a lot as attainable.
Lately, each time I carry out a recurring process–or see another person carry out one–I take into consideration
how we’d remove the human’s involvement completely by introducing brokers.
This each has labored effectively, but additionally labored poorly, and I wished to focus on the sample
I’ve discovered helpful.

For a concrete instance, an issue that each one software program firms have is patching safety vulnerabilities.
We have that downside too, and I test our safety dashboards periodically to make sure nothing has gone awry.
Sometimes once I test that dashboard, I’ll discover a discovering that’s precariously near our decision SLAs,
and both repair it myself or observe down the suitable staff to repair it.
However, this appears like a course of that shouldn’t require me checking on it.

Five to 6 months in the past, I added Github Dependabot webhooks as an enter into our inner agent framework.
Then I arrange an agent to deal with these webhooks, together with filtering incoming messages all the way down to the best precedence points.
About a month in the past, once I upgraded from GPT 4.1 to GPT 5.4 with excessive reasoning, I seen that it received
fairly good at utilizing the Github MCP to find out the suitable house owners for a given difficulty, utilizing the identical selection
of methods {that a} human would use: Codeowners recordsdata the place accessible, latest commits on
the repository, and so forth. The alerts and house owners have been already getting piped right into a Slack channel.

So, this labored! However, it didn’t truly work that effectively, as a result of regardless of repeated iteration on the immediate,
together with quite a few CRITICAL: you have to... statements, it merely couldn’t reliably limit itself to crucial
severity alerts. It would additionally embody some excessive severity alerts, and even the occasional medium severity alert.
This is a recurring difficulty with utilizing brokers as drop-in software program alternative: they merely are usually not good, and interrupting
your colleagues requires a degree of near-perfection.

If I’d employed somebody on our Security staff to inform groups about crucial alerts, and so they often flagged non-critical alerts,
ultimately somebody would pop into my DMs to ask me what was going incorrect. That didn’t occur right here, as a result of the information that these DMs
would present up prevented me from rolling the notifications out extra aggressively.
Coding brokers deal with this type of difficulty by operating checks, typechecking, or linting, however much less structured duties
are both tougher or dearer to confirm. For instance, I may have added an eval verifying messages didn’t point out medium or excessive
severity duties earlier than permitting it to ship to Slack, however I discovered that considerably unsatisfying regardless of understanding that it might work.

Instead, after some procrastination on different duties, I lastly prompted Claude to replace this agent to depend on
a code-driven workflow the place flow-control is managed by software program by default, and solely
cedes management to an agent the place ideally suited.
That workflow appears to be like like:

1. A webhook is available in from Dependabot
2. Script extracts the severity and motion (e.g. is it a brand new difficulty versus a resolved difficulty),
   and filters out low precedence or non-actionable webhooks
3. The code packages the metadata into a listing of points and repositories
4. The code passes every repository-scoped bundle to an agent with our inner possession ability and
   the Github MCP to find out applicable of us to inform for every difficulty
5. The points and possession information are handed to a second agent that codecs them
   as a Slack message

This works 100% of the time, whereas nonetheless permitting us to depend on our inner possession ability to
decide the most certainly groups or people to inform for a given downside.
It’s now one thing I can rollout extra aggressively.

The fast quick observe was a weekly follow-up ping for open crucial points,
counting on the identical break up of deterministic and agentic behaviors.
The subsequent enchancment will likely be automating the technology of the vulnerability fixes,
such that the human involvement is simply reviewing the change earlier than it mechanically
deploys.
(We already do this for Dependabot generated PRs,
however in my expertise Dependabot can remedy an inexpensive subset of recognized points,
however removed from all of them.)

That is the sample that I’ve discovered efficient:

1. Prototype with agent-driven workflow till I get a really feel for the workflow and what’s tough about it
2. Refactor agent-driven management away, more and more counting on code-driven workflow for an increasing number of of the answer
3. End with a model that narrowly depends on brokers for his or her strengths (navigating ambiguous issues like figuring out code house owners)

This has labored effectively for just about each downside I’ve encountered. The end-result is quicker, cheaper, and extra maintainable.
It’s additionally an inexpensive transition, usually I can take logs of some latest runs, the agent’s immediate, and a few transient directions,
throw them into Codex/Claude, and get a working alternative in a couple of minutes.