Self-reinforcing quality / adopting verification pattern

After implementing the small-change pattern, Imprint Engineering’s next focus
in improving our development loop has been
implementing a verification loop for our software development. This is an increasingly common
pattern, particularly popularized by poteto’s pstack,
but you’ll find similar ideas emerging in other places as well, e.g. Niko Matsakis’ post on
the correctness ratchet,
among others.

This captures some of the confusion I’ve seen in our initial rollout,
and the particulars of how we are implementing the approach.

What verification pattern is, and what it isn’t

The verification pattern suffers from confusing naming:
when you tell a team that you’re planning to roll it out,
they’ll often ask you what mechanism you’re using to verify the codebase,
don’t we already have tests, linters and so on?

However, the verification pattern itself is more similar to the
chain-of-thought pattern,
which aims to steer the harness towards your desired outcome in a clear way.
For example, we require that each pull request generate a verifications/{pr_num}.md file
that describes each step the harness took to verify the changes, and also what
it was unable to verify.

You can argue that your harness was already performing these steps, but it’s actually
hard to say whether it was or wasn’t. My sense is that it often wasn’t happening: I’ve seen too many pull requests go to CICD that
had failed basic lints that could have been run locally to believe the harness was always running them.
Now the harness has to explicitly document that it’s done those steps, or acknowledge it hasn’t.
This allows the harness to self-drive more effectively.

Equally importantly, this generates a renewing source of data about where your harness
is struggling to verify its work. That list is your roadmap for improving the ability
of your harness to self-drive work more reliably over time.

Our verification loop

The core of our verification loop is five steps:

1. Implement something that you’ve identified or that has been identified by an automated system to resolve.
   Let’s say addressing a new error that’s shown up in the logs.
2. Create a pull request containing that work after your harness has verified it locally.
   Pull requests cannot be moved from draft to ready until they have a verifications/{pr}.md verification file available
3. Provide feedback on the pull request, at minimum from our ai-code-review tool that automatically reviews all
   pull requests, but potentially also from humans
4. Incorporate and reject feedback on the pull request. For low quality feedback, we explicitly dismiss it using a /dismiss {reason}
   syntax in the pull request comment responses
5. Maintain our systems on a daily basis by pulling in the above steps to improve our software:
   1. Use pstack’s /maintain-verification-skill skill
      to update the feature maps and tools used based on what has changed and where verifications have run into issues.
      Automatically generate a pull request with these changes for review
   2. Review all verification files to identify where they struggled to verify, and generate issues
      for review that would reduce those gaps
   3. Review which feedback from ai-code-review was incorporated and which was rejected, tightening up its guidance

This is working pretty well for us so far, typically each of our major code areas (frontend, backend,
data, and native) wakes up with 4-6 pull requests to review as output of this process, improving our
codebase’s correctness while we sleep.

The above feedback loop still misses meta-issues in our CICD and testing patterns.
To address that, we’ve implemented the feedback tool pattern described by
Teddy Riker in Designing for Agents,
which has worked very well for us with respect to addressing places where CICD and harnesses themselves are
failing above the level of verifying a specific pull request.

feedback and verifications are complementary, not competing, techniques.
For example, if the container’s runner crashed before completing, the feedback tool would often
report on this issue, whereas the verification file wouldn’t even get generated.

Explore (raise ceiling) vs exploit (raise floor)

One of the fundamental tradeoffs in how organizations approach AI adoption comes down
to an explore vs exploit tradeoff. When you explore by having the full team do its own
thing, you explore for ideas which can raise the ceiling, but when you centralize you
raise the floor by ensuring everyone gets better PR review and better tooling.

The other related–and extremely open–question is whether exploration tends to over-optimize
for the particulars of a given individual or team’s preferences, which often reduces
cohesion across a codebase.
At this point, I think of misaligned guidance as “context poison”, and am pushing hard to
eliminate duplicate guidance on any topic, to ensure we have consistent, cohesive direction.

So far, our approach is running this as a centralized process for our repositories,
building a single, continually improving, corpus for each area using the self-reinforcing
loop built around the verification pattern. I think it’s working well for us, although
certainly I’ll admit that there’s some skepticism about how far we go towards centralizing,
and I think that skepticism is warranted: we do want to continue exploring, even if we
are primarily exploiting. The best techniques for AI-enhanced development in 2027 will be ideas
that are mostly unknown today.